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APPLICATION FOR PATENT TERM EXTENSION UNDER 35 U.S.C. §156 

Mail Stop: Patent Ext. 
Commissioner for Patents 
P.O. Box 1450 
Alexandria, VA 22313-1450 

Dear Sir: 

MELA Sciences, Inc. hereby applies for extension of the term of U.S. Patent No. 
6,208,749 to Gutkowicz-Krusin et al. (hereinafter "the '749 Patent"). 

Applicant MELA Sciences, Inc. is the owner of the '749 Patent by virtue of an 
assignment from all of the inventors to Electro-Optical Sciences, Inc. (recorded on April 24, 
1998 at Reel 009137, Frame 0244), and a subsequent name change of Electro-Optical Sciences, 
Inc. to MELA Sciences, Inc. (recorded on May 7, 2010 at Reel 024351, Frame 0849). 
Documentation of the foregoing may be found in Exhibit A and Exhibit B, respectively. 

MELA Sciences, Inc., acting through its duly authorized attorney, hereby submits this 
application for extension of patent term in accordance with 35 U.S.C. §156 and 37 C.F.R. 
§§1.710 - 1.785. The original and four copies of this application are being submitted. If any 
further information is required, please advise the undersigned. 

I. ELIGIBILITY FOR PATENT TERM EXTENSION 

Applicant believes the '749 Patent is eligible for patent term extension in accordance 
with 35 U.S.C. §156(a)(l)-(5) and 37 C.F.R. §1.720(a)-(h), as follows: 

A. The '749 Patent claims a product and a method of using or manufacturing the 
product as defined in 37 C.F.R. § 1.710, namely the MelaFind® system and methods for its use 
(37 C.F.R. § 1.710(b)(3) and §1. 720(a)). 

B. The term of the '749 Patent has never been previously extended, has not been 
extended under subsection (e)(1) of 35 U.S.C. §156, and has not been issued extensions under 37 
C.F.R. §§ 1.701, 1.760, or 1.790 (35 U.S.C. § 156(a)(2) and 37 C.F.R. § 1.720(b)). 
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C. This application for extension of the '749 Patent is submitted in compliance with 
35 U.S.C. §156(d)(l)-(4) and 37 C.F.R. §1.740 by the patent owner, MELA Sciences, Inc., 
through its undersigned representative (35 U.S.C. § 156(a)(3) and 37 C.F.R. § 1.720(c)). 

D. The product claimed in the '749 Patent, namely the MelaFind® system, has been 
subject to a regulatory review period as defined in 35 U.S.C. § 156(g) before its commercial 
marketing or use (35 U.S.C. § 156(a)(4) and 37 C.F.R. § 1.720(d)). 

E. The product claimed in the '749 Patent has received permission for commercial 
marketing or use after such regulatory review period, and such permission is the first received 
permission for commercial marketing or use under the provision of law under which such 
regulatory review occurred (35 U.S.C. §156(a)(5)(A) and 37 C.F.R. § 1.720(e)(1)). 

F. This application for extension of the '749 Patent is being submitted within the 
sixty-day period beginning on the date the product first received permission for commercial 
marketing or use under the provisions of law under which the applicable regulatory review 
period occurred (37 C.F.R. § 1.720(f)). The date the product first received permission is 
November 1, 201 1. As the sixty-day period ends on Saturday, December 31 st , and Monday, 
January 2 nd is New Year's Day observed, the sixty-day period ends on the next business day, i.e., 
January 3, 2012 (MPEP §2753). 

G. The term of the '749 Patent, including any interim extension issued pursuant to 37 
C.F.R. §1.790, has not expired before the submission of this application in compliance with 37 
CFR §1.741 (35 U.S.C. §156(a)(l), 37 C.F.R. §1.720(g)). 

H. No other patent term has been extended for the same regulatory review period for 
this product (37 C.F.R. § 1.720(h)). 

II. FORMAL REQUIREMENTS FOR APPLICATION 

A. Complete identification of the approved product as by appropriate chemical 

AND GENERIC NAME, PHYSICAL STRUCTURE OR CHARACTERISTICS (37 C.F.R. § 1 .740(a)(1)) 

The approved product is a device for characterizing the condition of a region of interest 
of skin, namely, the differentiation of pigmented skin lesions that are positive for melanoma 
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from those that are negative for the disease. The product is marketed under the trademarked 
name "MelaFind®." The MelaFind system is intended for use on clinically atypical cutaneous 
pigmented lesions with one or more clinical or historical characteristics of melanoma, excluding 
those with a clinical diagnosis of melanoma or likely melanoma. The MelaFind system is 
designed to be used when a dermatologist chooses to obtain additional information for a decision 
to biopsy; however, the device should not be used to confirm a clinical diagnosis of melanoma. 

The MelaFind system, depicted below in Figure 1, includes a handheld device for image 
capture, a base computer connected to the hand-held device via a universal serial bus cable, a 
touch-screen display monitor, and removable media. A high-level flowchart of the functionality 
of the MelaFind system is provided in Figure 3. The handheld imaging device, shown below in 
Figure 2 and flowcharted in Figure 4, is used to capture lesion images and includes an 
illuminator that provides light of ten specific wavelengths, a lens system that creates images of 
the light scattered back from the lesions, and a CMOS photo sensor. The base computer contains 
a processor that executes computer software having automatic image analysis algorithms. These 
algorithms include image calibration for noise and artifact reduction, image quality control for 
determining whether a lesion should be reimaged, lesion segmentation for identifying portions of 
the image associated with a lesion, feature extraction for computing parameters characterizing 
lesions, and lesion classification for evaluating whether a lesion exhibits the characteristics of 
melanoma. The sequence of these algorithms is further illustrated below in Figure 5. The touch- 
screen display allows the operator to examine the visual output of the MelaFind system and 
otherwise interact with the device. 
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Handheld Imaging 
Device 



Figure 1 - MelaFind® System 




Figure 2 - Handheld Imaging Device 
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Figure 3 - MelaFind System High-Level Functionality Flowchart 
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Figure 4 - Imager Flowchart 
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Figure 5 - Algorithms Flowchart 
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B. Complete identification of the Federal statute including the applicable 

PROVISION OF LAW UNDER WHICH THE REGULATORY REVIEW OCCURRED (37 C.F.R. § 1 .740(a)(2)) 

The approved device, the MelaFind system, was subject to regulatory review under 
Section 515 of the Federal Food, Drug, and Cosmetic Act (21 U.S.C. §360e) as a medical device, 
and was subject to a binding protocol agreement under Section 520(g)(7) of the Federal Food, 
Drug, and Cosmetic Act (21 U.S.C. §360j(g)(7)). The approved device was clinically tested as a 
not significant risk device under 21 C.F.R. §8 12.2(b) in accordance with Section 520(g) of the 
Federal Food, Drug, and Cosmetic Act (21 U.S.C. §360j). 

The regulatory review was conducted by the Center for Devices and Radiological Health 
(CDRH) of the United States Food and Drug Administration (FDA), including CDRH's Office 
of Device Evaluation - Division of Surgical, Orthopedic and Restorative Devices. 

C. Identification of the date on which the product received permission for 

COMMERCIAL MARKETING OR USE UNDER THE PROVISION OF LAW UNDER WHICH THE APPLICABLE 
REGULATORY REVIEW PERIOD OCCURRED (37 C.F.R. § 1 .740(a)(3)) 

The product received permission for commercial marketing or use under Section 515(d) 
of the Federal Food, Drug, and Cosmetic Act (21 U.S.C. §360e(d)) on November 1, 201 1. 

D. Statement that the application is being submitted within the sixty day period 

PERMITTED FOR SUBMISSION PURSUANT TO § 1 .720( F) AND AN IDENTIFICATION OF THE DATE OF 
THE LAST DAY ON WHICH THE APPLICATION COULD BE SUBMITTED (37 C.F.R. § 1 .740(a)(5)) 

The present application for extension is being submitted within the sixty-day period 
permitted for submission pursuant to 37 C.F.R. § 1.720(f). The last day on which this application 
may be submitted is December 31, 201 1. However, since December 31 st is a Saturday, and 
Monday, January 2 nd is New Year's Day observed, the last day on which this application may be 
submitted is January 3, 2012. 
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E. Complete identification of the patent for which an extension is being sought by 

THE NAME OF THE INVENTOR, THE PATENT NUMBER, THE DATE OF ISSUE, AND THE DATE OF 

expiration (37 C.F.R. 81.740(a)(6)) 

The patent for which an extension is being sought is U.S. Patent No. 6,208,749 Bl, which 
issued March 27, 2001. The inventors are Dina Gutkowicz-Krusin, Marek Elbaum, Michael 
Greenebaum, and Adam Jacobs. The '749 patent issued on a continued prosecution application 
filed under 37 C.F.R. § 1.53(d), and is subject to the twenty year patent term provisions of 35 
U.S.C. § 154(a)(2). The '749 Patent is not subject to any terminal disclaimer. The term of the 
'749 patent was adjusted under 35 U.S.C. § 154(b) by zero (0) days. The '749 patent is therefore 
set to expire twenty years from its filing date of February 27, 1998, i.e., February 27, 2018, 
subject to payment of maintenance fees. 

F. Copy of the patent for which an extension is being sought, including the entire 

SPECIFICATION (INCLUDING CLAIMS) AND DRAWINGS (37 C.F.R. § 1.740(a)(7)) 

A copy of the entire '749 Patent is attached as Exhibit C. 

G. Copy of any disclaimer, certificate of correction, receipt of maintenance fee 

PAYMENT, OR REEXAMINATION CERTIFICATE ISSUED IN THE PATENT (37 C.F.R. § 1 .740(a)(8)) 

No disclaimer or reexamination certificate has been filed or issued in the '749 Patent. 
Copies of the applicable maintenance fee payment receipts from September 9, 2004 and April 
14, 2008 are attached as Exhibit D. The twelfth-year maintenance fee window will not open 
until March 27, 2012. No certificate of correction has been issued for the '749 Patent; however, 
a Request for Certificate of Correction has been filed concurrently with this application to correct 
a minor typographical error in independent method claim 17. A copy of the request can be found 
in Exhibit E. 
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H. Statement that the patent claims the approved product, or a method of using 

OR MANUFACTURING THE APPROVED PRODUCT, AND A SHOWING WHICH LISTS EACH APPLICABLE 
PATENT CLAIM AND DEMONSTRATES THE MANNER IN WHICH AT LEAST ONE SUCH PATENT CLAIM 
READS ON THE APPROVED PRODUCT AND THE METHOD OF USING THE APPROVED PRODUCT (31 
C.F.R.S 1.740(a)(9)) 

The '749 patent claims the approved MelaFind product and a method of using the 
approved MelaFind product. Applicant asserts that at least claims 45-68 read on the approved 
product, and that at least claims 1-9, 11,15, 17-20, 22, 26-29, 32-39, 40 and 43 read on a method 
of using the approved product. 

Independent system claim 45 reads on the approved product, as follows: 



Table 1 - Chart for Cla 


im 45 of the '749 Patent 


Claim 45 of the '749 Patent 


Corresponding Features in the MelaFind® 
Svstem 


A system for characterizing the condition of a 
region of interest of skin, comprising: 


The MelaFind© system, shown in Figure 1, is 
a multi-spectral, non-invasive computer-vision 
system that classifies the image of a pigmented 
skin lesion based upon degree of 
disorganization: MelaFind positive (high 
degree of disorganization) or MelaFind 
negative (low degree of disorganization). The 
MelaFind system is intended for use on 
clinically atypical pigmented skin lesions with 
one or more clinical or historical characteristics 
of melanoma and is designed to be used when 
a dermatologist chooses to obtain additional 
information for a decision to biopsy. 


a source of illumination of light in at least three 
spectral bands; 


Figure 2 illustrates the MelaFind handheld 
imager, that includes an LED illuminator 
emitting light in ten spectral bands, from 430 
to 950 nm. 



LIBC/4215148.3 



10 



Express Mail No. .: EV 951019703 US 



Patent Term Extension Application for U.S. Pat. No. 6,208,749 



Claim 45 of the '749 Patent 


Corresponding Features in the MelaFind® 
Svstem 


a camera for acquiring digital images of the 
region of interest based on the light re-emitted 
from the illuminated region of interest at each 
of the spectral bands, the digital image 
comprising digital signals whose values are a 
function of the condition of the region of 
interest; 


Figure 2 shows the camera within the 
MelaFind handheld imager, including a 9- 
element Zeiss lens and a CMOS photo sensor. 
The camera is used for acquiring digital images 
of a region of interest that includes an atypical 
pigmented skin lesion based on the light re- 
emitted from the illuminated region of interest 
in each of ten spectral bands. The values of 
intensity in these images are a function of the 
condition (i.e., the degree of disorganization) 
in the region of interest. 


memory for storing the digital images provided 
by the camera; 


Figure 1 shows that the MelaFind system's 
base computer incorporates memory 
(electronic storage media) for storing the 
digital images provided by the camera. 


a digital processor programmed to perform the 
steps of: 


Figure 1 shows the base computer having a 
digital processor. The processor is 
programmed to perform the analysis of multi- 
spectral images. 


segmenting the digital images stored in 
memory by generating a single segmentation 
mask, where the single segmentation mask is 
the segmentation mask having largest area of 
segmentation masks generated from each 
image in each of the at least three spectral 
bands; 


Figure 5 illustrates the algorithmic functions 
performed by the MelaFind system. Following 
image calibration and quality checking, the 
digital processor executes programming that 
segments images to identify an atypical 
pigmented skin lesion in the field of view 
("Segmentation"). A single segmentation mask 
is created for all ten spectral bands; this is the 
mask having the largest area of the 
segmentation masks generated in each spectral 
band. 


estimating at least one value for each digital 
image at each spectral band which is a function 
of the texture of the portion of the region of 
interest determined by the segmentation mask; 


Figure 5 further shows that the MelaFind 
svstem estimates values in each snectral band 
("Parameter Estimation") as part of the 
sequence of executed algorithms. The values 
are functions of the texture of the region of 
interest determined by the segmentation mask. 


characterizing the condition of the skin based 
on the estimated values; and 


As shown in Figure 5, the MelaFind processor 
characterizes the condition of the skin as 
having a high or low degree of disorganization 
based on estimated values (75 parameters) in 
all ten spectral bands ("Classification"). 
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Claim 45 of the '749 Patent 


Corresponding Features in the MelaFind® 
System 


outputting the characterization of the region of 
interest. 


As depicted in Figure 3, following the 
targeting, imaging, and algorithm steps, the 
MelaFind system outputs the characterization 
of the region of interest to the user: MelaFind 
positive (high degree of disorganization) or 
MelaFind negative (low degree of 
disorganization). 


Independent method claim 17 reads on a method 

Table 2 - Chart for Cla 


for using the approved product, as follows: 
im 17 of the '749 Patent 


Claim 17 of the' 749 Patent 


Corresponding Features in the MelaFind® 
System 


A method of characterizing the condition of a 
region of interest of skin, wherein the 
absorption and scattering of light in different 
spectral bands by the region of interest is a 
function of the condition of the skin, the 
method comprising: 


The MelaFind® system, shown in Figure 1, is 
a multi-spectral, non-invasive computer-vision 
system that classifies the image of a pigmented 
skin lesion based upon degree of 
disorganization: MelaFind positive (high 
degree of disorganization) or MelaFind 
negative (low degree of disorganization). The 
MelaFind system is intended for use on 
clinically atypical pigmented skin lesions with 
one or more clinical or historical characteristics 
of melanoma and is designed to be used when 
a dermatologist chooses to obtain additional 
information for a decision to biopsy. 
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Claim 1 7 of the .'749 Patent 


Corresponding Features in the MelaFind® 
Svstem 


illuminating a portion of the skin including the 
rpcnnn of inrprp^t hv lic^ht in at 1pa<it thrpp 

spectral bands; 


Figure 2 illustrates the MelaFind handheld 

itnaffpr that inplnHpQ an T T^Tl illiiminatnr iicpH 

to illuminate a portion of skin by emitting light 
in ten spectral bands, from 430 to 950 nm. 
Figure 4 shows the imaging process, starting 
with the illumination of the region of interest. 


digitally imaging the portion of the skin 
including the region of interest at the at least 
tree 1 spectral bands with the light re-emitted by 
the portion of the skin to generate digital 
images comprising digital signals whose values 
are a function of the condition of the region of 
interest of the skin; and 


Figure 2 shows a camera within the MelaFind 
handheld imager, including a 9-element Zeiss 
lens and a CMOS photo sensor. The camera is 
used for digitally imaging a region of interest 
that includes an atypical pigmented skin lesion 
based on the light re-emitted from the 
illuminated region of interest in each of ten 
spectral bands. The values of intensity in these 
images are a function of the condition (i.e., the 
decree of disorganization^ in the region of 
interest. Figure 4 shows a flowchart of the 
imaging process, in which the lens and sensor 
facilitate the digital imaging of the illuminated 
region of interest. 


providing the digital images to a processor, 
wherein the processor: 


Figure 1 shows the base computer having a 
digital processor. The processor is 
programmed to perform the analysis of multi- 
spectral images. 


segments the digital images by generating a 
single segmentation mask defining the 
boundary of the region of interest for each 
image, where the single segmentation mask is 
the segmentation mask having largest area of 
segmentation masks generated from each 
image in each of the last least three spectral 
bands; 


Figure 5 illustrates the algorithmic functions 
performed by the MelaFind system. Following 
image calibration and quality checking, the 
digital processor executes programming that 
segments images to identify an atypical 
pigmented skin lesion in the field of view 
("Segmentation"). A single segmentation mask 
is created for all ten spectral bands; this is the 
mask having the largest area of the 
segmentation masks generated in each spectral 
band. 



1 Applicant recognizes that a typographical error exists in this claim 17. The correct term should be "three." 

Applicant has submitted a Request for Certificate of Correction concurrently with this application, a copy of 
which is provided in Exhibit E. 
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Claim 17 of the < 749 Patent 


Corresponding Features in the MelaFind® 
Svstem 


computes at least one estimated value for each 
digital image at each spectral band which is a 
function of a characteristic of the region of 
interest determined by the segmentation mask; 


Figure 5 further shows that the MelaFind 
system estimates values in each spectral band 
("Parameter Estimation") as part of the 
sequence of executed algorithms. The values 
are functions of the texture of the region of 
interest determined by the segmentation mask. 


characterizes the condition of the region of 
interest of the skin based on the estimated 
values; and 


As shown in Figure 5 the MelaFind nrocessor 

/ *■ LJ ij A A V/ Tf 11 111 A A V»-l V % \fA A \J X * A W J. K+A. AAA V-J- A \J v V <J V/ A 

characterizes the condition of the skin as 
having a high or low degree of disorganization 
based on estimated values (75 parameters) in 
all ten spectral bands ("Classification"). 


outputs the characterization of the condition of 
the region of interest of the skin. 


As depicted in Figure 3, following the 
targeting, imaging, and algorithm steps, the 
MelaFind system outputs the characterization 
of the region of interest to the user: MelaFind 
positive (high degree of disorganization) or 
MelaFind negative (low degree of 
disorganization). 



— end of page — 
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I. Relevant dates and information pursuant to 35 U.S.C. § 156(g) in order to 

ENABLE THE SECRETARY OF HEALTH AND HUMAN SERVICES TO DETERMINE THE APPLICABLE 
REGULATORY REVIEW PERIOD (37 C.F.R. § 1 J4QU)( 10)) 

The '749 Patent claims a medical product that was subject to an approved investigational 
device exemption (IDE) as a not significant risk (NSR) device by the FDA, pursuant to 21 C.F.R. 
§812.2(b); therefore, 37 C.F. R. § 1.740(a)(10)(v) applies. 

(A) The effective date of the investigational device exemption (IDE) and the IDE number, 
if applicable, or the date on which the applicant began the first clinical investigation involving 
the device, if no IDE was submitted, and any available substantiation of that date: 

The effective date on which the applicant obtained approval of a not significant risk IDE 
to conduct clinical investigations, i.e., the Institutional Review Board (IRB) approval date, was 
May 2, 2001 (see Exhibit F for the Certificate of Approval). Applicant believes the effective 
date of the first clinical investigation involving the device is the IRB approval date for the 
purposes of this application. See 21 C.F.R. §60.22(c)(l)(i) and (ii). 

(B) The date on which the application for product approval or notice of completion of a 
product development protocol under Section 515 of the Federal Food, Drug and Cosmetic Act 
was initially submitted and the number of the application: 

The application for product approval under Section 515 of the Federal Food, Drug, and 
Cosmetic Act was initially submitted and filed by the FDA on June 9, 2009. The letter from the 
FDA Center for Devices and Radiological Health provided in Exhibit G substantiates this date. 
The PMA number was P090012. 

(C) The date on which the application was approved or the protocol declared to be 
completed: 

The PMA was approved on November 1, 201 1. The letter from the FDA substantiating 
that approval date is provided in Exhibit H. 

- end of page - 
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J. Brief description of the significant activities undertaken by the marketing 

APPLICANT DURING THE APPLICABLE REGULATORY REVIEW PERIOD WITH RESPECT TO THE 
APPROVED PRODUCT AND THE SIGNIFICANT DATES APPLICABLE TO SUCH ACTIVITIES (37 C.F.R. 

§1.740(a)aiV) 

Attached as Exhibit I is a table listing significant activities undertaken by the marketing 
applicant during the applicable regulatory review period with respect to the approved product 
and the dates such activities occurred. Further details regarding the marketing applicant's 
activities during the regulatory review period may be provided upon request. 

— end of page — 
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K. Statement that in the opinion of the applicant the patent is eligible for the 

EXTENSION AND A STATEMENT AS TO THE LENGTH OF EXTENSION CLAIMED, INCLUDING HOW THE 
LENGTH OF EXTENSION WAS DETERMINED (37 C.F.R. $1.740(aX12V) 

In the opinion of the Applicant, U.S. Patent No. 6,208,749 is eligible for patent term 
extension, under 35 U.S.C. § 156 because: 

(i) one or more of the claims of U.S. Patent No. 6,208,749 claim the approved 
product (the MelaFind system); 

(ii) the term of U.S. Patent No. 6,208,749 has not been extended on the basis of 35 
U.S.C. § 156 before the submission of the instant application; 

(iii) the term of no other U.S. Patent has been extended under 35 U.S.C. § 156 on the 
basis of the regulatory review process associated with the approved product ( the MelaFind 
system); 

(iv) there is an eligible period of regulatory review by which the patent may be 
extended pursuant to35U.S.C.§156; 

(v) the present application has been submitted within the 60-day period following the 
approval date of the approved product, pursuant to 35 U.S.C. § 156(c); 

(vi) the applicant before the FDA is the owner of U.S. Patent No. 6,208,749 by virtue 
of the assignment documents attached as Exhibit A and Exhibit B; and 

(vii) the application submitted here otherwise complies with all requirements of 35 
U.S.C. § 156 and all applicable rules and procedures. 

The period of extension of the term of U.S. Patent No. 6,208,749 requested by Applicant 
is 5 (five) years, such that the patent would expire on February 27, 2023. The length of the 
extension was determined according to 35 U.S.C. §§ 156(c), (g)(3), (g)(6), and 37 C.F.R. §1.777, 
as shown below. 
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Patent Term Extension Application for U.S. Pat. No. 6,208,749 

1) The number of days in the period beginning on the date a clinical investigation on 
humans involving the device was begun and ending on the date an application was 
initially submitted with respect to the device under section 5 1 5 of the Federal Food, 
Drug, and Cosmetic Act is 2960 days. 

2) The number of days in the period beginning on the date the application was initially 
submitted with respect to the device under section 515 of the Federal Food, Drug, and 
Cosmetic Act, and ending on the date such application was approved under such Act or 
the period beginning on the date a notice of completion of a product development 
protocol was initially submitted under section 515(f)(5) of the Act and ending on the date 
the protocol was declared completed under section 515(f)(6) of the Act is 875 days. 

3) The dates pertaining to nos. 1 and 2 above, so that it can be determined how many days 
occurred prior to issuance of the patent, are as follows. The IRB approval date was May 
2, 2001. The PMA submission date was June 9, 2009. U.S. Patent No. 6,208,749 issued 
on March 27, 2001, whereby 0 days should be subtracted based on activities on or before 
the patent issuance date. 

Applicant submits that it has acted with due diligence throughout the clinical 
investigation period and the pre-market approval period, whereby 0 days should be 
subtracted from the regulatory review period for lack of due diligence. 

One half the number of days remaining in the period defined by paragraph (c)(1) of 37 
C.F.R. §1.777: 1480 days 

Total relevant period pursuant to 37 C.F.R. § 1.777(d)(1): 1480 + 875 = 2355 days. 

4) The expiration date of the patent: 

Pursuant to 37 C.F.R. § 1.777(d)(5): Adding 5 years to the original expiration of the 
patent, the new expiration date of the patent appears to be February 27, 2023, which is 
the earlier date, as compared to adding 2355 days to the original expiration date, and is 
not greater than 14 years from the date of regulatory approval, November 1, 201 1. 
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L. Statement that applicant acknowledges a duty to disclose (37 C.F.R. 
§1.740(a)(13)) 

Applicant acknowledges a duty to disclose to the Director of the United States Patent and 
Trademark Office and the Secretary of Health and Human Services any information which is 
material to the determination of entitlement to the extension sought herein. 

M. Prescribed fee for receiving and acting upon the application for extension (37 
C.F.R. §1.740(a)(14V) 

Please charge the fee prescribed in 37 C.F.R. §120(j)(l) for a patent term extension 
application under 35 U.S.C. §156 to Deposit Account No. 07-1700. The Director is authorized 
to charge the required fees and any underpayment or credit any overpayment to Deposit Account 
No. 07-1700 for any necessary matter in connection with this application. 

N. Name, address, and telephone number of the person to whom inquiries and 
correspondence relating to the application for patent term extension are to be 
DIRECTED (37 C.F.R. §1.740(a)(15)) 

Correspondence in connection with this application shall be directed to: 

Christopher W. Stamos 
c/o Patent Administrator 
Goodwin Procter LLP 
Exchange Place 
53 State Street 
Boston, MA 02109 
Phone: 617.570.1000 
Fax: 617.523.1231 
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Date: / 7 Xk^W^ By: 



Attorneys for Applicant 
GOODWIN PROCTER LLP 
Exchange Place 
53 State Street 
Boston, MA 02109 
Phone: 617.570.1000 
Fax: 617.523.1231 
Customer No. 51414 



Respectfully submitted, 




Christopher W. Stamos 
USPTO Reg. No. 35,370 
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Please record the attached original document or copy thereof 
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Systems and methods for the multispectral imaging of skin 
tissue enables automatic characterization of the condition of 
a region of interest of the skin, based on direct digital 
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parameters. The region of interest may include a skin lesion, 
in which case the present invention enables the character- 
ization of the lesion as malignant or benign. 
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SYSTEMS AND METHODS FOR THE 
MULTISPECTRAL IMAGING AND 
CHARACTERIZATION OF SKIN TISSUE 

This application claims the benefit of U.S. provisional 5 
application Ser. Nos, 60/039,218 and 60/039,407, both of 
which were filed on Feb. 28, 1997, and are incorporated by 
reference, herein. 

STATEMENT REGARDING FEDERALLY io 
SPONSORED RESEARCH 

This invention was made with Government support under 
NIH/National Cancer Institute Contract No. 2-R44- 
CA60229-02A1 and U.S.A.F. Phillips Laboratory Contract 
No. F29601-95-C-0125. The Government has certain rights 
in this invention. 

FIELD OF THE INVENTION 

This invention relates to methods and systems for the 20 
computer controlled image analysis of digital skin tissue at 
a plurality of wavelengths, which may include those outside 
of the red-green-blue bands. The methods and systems 
further include the automatic characterization of the condi- 
tion of the skin tissue, based on automatically computed 25 
values of parameters which are functions of characteristics 
of the skin tissue, based on the digital images. Skin lesions 
can be analyzed for determining whether the lesion is a 
melanoma, for example. Systems for digitally imaging and 
analyzing skin tissue are disclosed, as well. 30 

BACKGROUND OF THE INVENTION 

Melanoma is a usually fatal skin cancer, unless it is 
detected and surgically removed in its earliest stages. Early 
detection of malignant melanoma is difficult because early 
melanomas, those having a Breslow thickness less than 1 
mm, share many diagnostic features with benign lesions, 
such as dysplastic nevi or atypical melanocytic nevi. 

To aid in the analysis of lesions, conventional 40 
photography, referred to as "clinical imaging", has been 
used to image the lesion for further study. The effectiveness 
of clinical imaging can be compromised, however, by specu- 
lar reflection by the skin. Polarizers have been used for 
polarized imaging, which minimizes specular reflection. 45 

Dermoscopy is another technique for examining skin, in 
which specular reflection is minimized. Dermoscopy also 
assists in clinically differentiating melanoma from its benign 
simulants by enabling the observation of features of pig- 
mented melanocytic lesions that are not discernible by the 50 
naked eye. In dermoscopy, the skin is made more transparent 
to light by providing an oil layer over the skin, in front of the 
optical system. A glass plate is placed over the oil layer. The 
oil has an index of refraction between the index of refraction 
of the horny layer of the skin and the glass plate. Standard 55 
magnifying optics may be used to enlarge the structures 
rendered visible on and under the surface of the skin by the 
oil layer. The region of interest can then be examined 
visually. Slides of the region of interest can be made, as well, 
for future study. 60 

Despite their similarities, most malignant melanomas 
differ in certain of their characteristics from other melano- 
cytic lesions. A major advance in characterizing skin lesions 
based on certain of the observable differences between 
malignant and other lesions is the "ABCD" rule, where 65 
A=asymmetry, B^border irregularity, C=color variability, 
and D^diameter greater than 6 mm. A corresponding ABCD 
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rule, where "D" refers to dermoscopic structures, such as 
brown globules, black dots or pigment networks within the 
lesion, is applied to dermoscopic images. Because the clini- 
cal and dermoscopic applications of these rules are 
subjective, they are not very reliable. 

When skin is illuminated by light, the light can be 
re-emitted by reflection, scattering or fluorescence. It is 
known in the art that the re-emission of light absorbed at 
different wavelengths by a region of interest of skin can 
provide different information. For example, as the wave- 
length of the light increases, its depth of penetration into the 
skin or other tissue also increases. Chromophores at different 
depths in the tissue therefore absorb and re-emit light at 
various wavelengths. Melanin and hemoglobin are examples 
of such chromophores. 

Since the unaided eye cannot perceive light outside of the 
visible region or low-contrast structure in visible-light 
images, information which may be useful in diagnosing a 
lesion may not be directly observable. Digital acquisition 
and processing of dermoscopic images may, therefore, 
improve diagnostic reliability by employing more of the 
information residing in such images that is not directly 
observable. There have therefore been attempts to use 
objective, computer-based, image analysis algorithms that 
can discern meaningful differences between benign and 
malignant melanocytic lesions with sufficient accuracy. 

Computer processing of images requires that the image be 
in digital form. A digital image is an array of digital signals 
whose values are a function of certain characteristics of the 
subject of the image. When imaging skin lesions, the digital 
images comprise digital signals whose values are a function 
of the re-emission characteristics of the skin and lesion, at 
different spectral bands of light. The array is obtained by 
spatial sampling and quantizing the intensity of images 
obtained with film or directly by electronic cameras. Prac- 
tical limitations on the number of picture elements or pixels 
per unit area of image determine the achievable spatial 
resolution of the digital image. The digital image typically 
needs to be segmented to separate the digital signals which 
are a function of the skin lesion from the digital signals 
which are a function of the surrounding skin. 

Computer aided analysis has also been used to classify 
skin lesions using quantitative values indicative of particular 
characteristics of lesions, referred to as parameters. Based 
on histopatholdgical diagnosis of lesions, algorithms have 
been developed which use linear or non-linear classifiers to 
combine parameters provided by the operator of an imaging 
device or a physician or computed by a processor, to yield 
a value which can be used to classify the lesion. Because 
some of the steps in the computer-aided analysis of which 
we are aware depend on subjective judgments of an 
individual, such analyses may provide highly variable 
results. 

The images heretofore available have been obtained with 
commercially available red-green-blue color imaging appa- 
ratus. Color photographic transparencies of skin lesions 
have been digitized and skin lesions have been directly 
imaged with "three-chip" digitizing cameras. Such cameras 
employ broad-band filter bandpasses that are ultimately 
based on the wavelength response of the human visual 
system and have large regions of overlap. 

Electronic images may also be obtained in narrower, 
non-overlapping filter bandpasses, which may reveal 
additional, wavelength-dependent differences between the 
images of melanomas and of benign lesions. However, such 
devices have had poor resolution and/or poor signal-to-noise 
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characteristics which prevent the acquisition of digital 
images of melanocyte skin lesions of sufficient quality for 
effective application of machine vision techniques for lesion 
diagnosis. 

Existing imaging systems and processes also tend to 5 
suffer from an inability to provide the required repeatability 
of the values of extracted lesion parameters, due in part to 
a lack of standardization with respect to spatially varying 
artifacts. The parameters, therefore, lack invariance to light- 
ing and image exposure conditions, for example. Obtaining 10 
high signal-to-noise ratios in images recorded in narrow 
filter bandpasses, when exposure times are sufficiently short 
that the skin is effectively "frozen" during the exposure 
sequence, has also been difficult. In addition, since the 
optimum wavelengths for automatic characterization may 15 
not be the optimum wavelengths for visual observation, it 
may be difficult to reconstruct high-fidelity color images 
from the digital images for visual interpretation by a clini- 
cian. 

The assessment of wounds and burns through the appear- 20 
ance of color images present similar challenges. Existing 
technology for the imaging of skin in vivo for these purposes 
is also inadequate. Practical solutions to the problems of 
employing multispectral digital imaging of skin for the 
analysis of lesions, wounds, or other conditions have not 25 
been found. 

SUMMARY OF THE INVENTION 

The methods and systems of the present invention provide 30 
for the acquisition of digital images of skin at a plurality of 
spectral bands to automatically characterize the condition of 
the tissue based on the digital images. Spectral wavelength 
bands within and outside of the visible band may be used. In 
accordance with the present invention, a pigmented skin 35 
lesion can be characterized as malignant or benign, for 
example. The digital images comprise a plurality of digital 
signals whose values are functions of the condition of the 
tissue. The digital images acquired are subjected to objective 
and quantitative analysis by a digital processor to detect and 40 
identify abnormalities. The analysis includes image 
segmentation, parameter estimation and characterization of 
the skin. The estimation and characterization steps are 
automatic. The segmentation step may be automatic, as well. 
Subjective judgments are therefore minimized or eliminated. 45 

It has been found that generating the segmentation mask 
from a digital image acquired with light in a spectral band 
which does not penetrate deeply into the skin, such as a blue 
spectral band, provides superior results. After segmentation, 
estimated values which are functions of characteristics of the 50 
lesion, such as its texture, asymmetry, blotchiness, and 
border irregularities, are computed and used to automati- 
cally characterize the condition of the skin. Digital signals 
corresponding to hair or blob-like structures are preferably 
removed during segmentation. 55 

In accordance with the present invention, a method for 
characterizing the condition of a region of interest of the 
skin, wherein the absorption and scattering of light in 
different spectral bands by the region of interest is a function 
of the condition of the skin, is disclosed. The method 60 
comprises illuminating the region of interest of the skin by 
light in at least three spectral bands and digitally imaging the 
region of interest at the at least three spectral bands with the 
light re-emitted by the skin to generate digital images 
comprising digital signals whose values are a function of the 65 
condition of the skin. The digital images are provided to a 
processor which segments the digital images by generating 
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a segmentation mask from a digital image in any one of the 
at least three spectral bands, computes at least one estimated 
value for each digital image at each spectral band which is 
a function of a characteristic of the region of interest within 
the segmentation mask, characterizes the condition of the 
skin based on the estimated values, and outputs the charac- 
terization of the condition of the skin. Preferably, the 
segmenting, estimating and characterizing steps are con- 
ducted without the intervention of an operator. Useful 
parameters include measures of the texture, asymmetry, 
blotchiness and border irregularity of the portion of the 
region of interest. 

The digital images may be obtained by directly imaging 
the region of interest with a digital camera, or digitally 
imaging color slides of the region of interest, through 
appropriately filtered light. 

The characterizing step may include comparing a 
weighted combination of the parameter values against a 
threshold value. The weight coefficients for each parameter 
value and the threshold value may be selected based on a 
training set of images of lesions or other skin conditions, 
whose condition has been determined, preferably through 
histological examination by a plurality of doctors. 
Preferably, specificity is maximized under the constraint of 
100% sensitivity to melanoma. 

In accordance with another aspect of the invention, a 
system for characterizing the condition of a region of 
interest of skin includes means for illuminating the region of 
interest with light in at least three spectral bands and a 
camera for acquiring digital images of the region of interest 
based on the light re-emitted from the illuminated region of 
interest at each of the spectral bands. The digital image 
comprises digital signals whose values are a function of the 
condition of the region of interest. A digital processor 
segments the digital images by generating a segmentation 
mask from a digital image in any one of the at least three 
spectral bands, and computes at least one estimated value for 
each digital image at each spectral band which is a function 
of the texture of the portion of the region of interest within 
the segmentation mask. The processor characterizes the 
lesion based on the estimated value or values. The other 
parameters discussed above may be used, as well. 

The camera may be a single-chip or multiple-chip charge- 
coupled device which detects light in a plurality of spectral 
bands between the near ultraviolet to near infrared. The filter 
means may be a plurality of interference filters mounted on 
a wheel for stepping any filter into a position intercepting the 
light from the light source. Preferably, at least one of the 
spectral bands has a center which lies between about 400 and 
500 nanometers, at least one of the spectral bands has a 
center which lies between about 500 and 600 nanometers, 
and at least one other spectral band has a center which lies 
between about 750 and 1000 nanometers. 

DESCRIPTION OF THE FIGURES 

FIG. 1(a) is a schematic illustration of a method and 
system of imaging a region of interest of skin in accordance 
with the present invention; 

FIG. 1(b) is a schematic illustration of a plurality of 
narrow spectral bandwidths which may be used to illuminate 
the skin in the embodiment of FIG. 1(a); 

FIG. 1(c) is a schematic illustration of alternative methods 
and systems for digitizing and analyzing color photographic 
slides of a region of interest of skin; 

FIG. 2 is a schematic illustration of preferred illumination 
and imaging portions of a computer controlled imaging 
system for direct imaging of a lesion; 
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FIG. 3(a) is a flow chart of a calibration procedure for use 
with the present invention; 

FIG. 3(6) is a flow chart of a method of processing images 
for classifying lesions as malignant or benign, in accordance 
with the present invention; 5 

FIGS. 4(a) and 4(6) are histograms of a malignant mela- 
noma and of an atypical melanocytic nevus, respectively, 
showing two peaks in each histogram; 

FIGS. 5(a) and 5(b) are histograms of another malignant 
melanoma and another atypical melanocytic nevus, 
respectively, showing three or more peaks in each histo- 
gram; 

FIGS. 6(a) and 6(d) are digital images in the blue spectral 
band of another malignant melanoma and another atypical 15 
melanocytic nevus, respectively; 

FIGS. 6(b) and 6(e) are digital images of the images of 
FIGS. 6(a) and 6(d) respectively, after thresholding; 

FIGS. 6(c) and 6(f) are digital images of the images of 
FIGS. 6(a) and 6(d), respectively, after iterative threshold- 20 
ing; 

FIGS. 1(a) and 1(d) are digital images in the blue spectral 
band of another malignant melanoma and another atypical 
melanocytic nevus; 

FIGS. 7(6) and 1(e) are digital images of FIGS. 1(a) and 25 
1(d), respectively, resulting from iterative processing and 
showing dark blobs outside the lesion area; 

FIGS. 7(c) and 1(f) are digital image masks of FIGS. 1(b) 
and 1(d), respectively, resulting from image cleaning; 30 

FIGS. 8(a) and 8(e) are digital images in the blue spectral 
band of another malignant melanoma and another atypical 
melanocytic nevus, respectively, showing hair; 

FIGS. 8(6) and 8(f) are reverse intensity contrast images 
of the lesions of FIGS. 8(a) and 8(e), respectively; 35 

FIGS. 8(c) and 8(g) are digital images resulting from an 
averaging process applied to the images of FIGS. 8(a) and 
8(6), to remove hair; 

FIGS. 8(d) and 8(h) are binary lesion masks resulting 4Q 
from the segmentation of the images of FIGS. 8(c) and 8(g), 
respectively; 

FIG. 9 is a spatial filter used to remove hair; 

FIGS. 10(a)-10(c) are segmented digital images in the 
blue, green and red spectral bands, of the malignant mela- 45 
noma whose histogram is shown in FIG. 5(a); 

FIGS. lQ(d)-10(f) are segmented digital images in the 
blue, green and red spectral bands, of an atypical melano- 
cytic nevus whose histogram is shown in FIG. 5(6); 

FIG. 11 is a chart of lesion parameters and their associated 50 
diagnostic accuracy, sensitivity and specificity when used 
individually; 

FIG. 12 is a plot of linear classifier values versus lesion 
identification number, for 41 malignant melanomas and 104 55 
atypical melanocytic nevi; 

FIG. 13 is a plot of linear classifier values versus lesion 
identification number for 24 superficial spreading melano- 
mas and 16 melanomas in-situ; and 

FIG. 14 is a plot of lesion parameter versus Breslow 60 
thickness for 24 superficial spreading melanomas. 

DESCRIPTION OF THE INVENTION 

FIG. 1(a) is a schematic illustration of a method and 
system 1 in accordance with the present invention, by which 65 
images of the skin 2 are acquired by a camera nearly 
simultaneously at a plurality of spectral bands, X„ i=l,2, . . . 
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M, that are preferably effectively non-overlapping, as shown 
schematically in FIG. 1(6). The skin is illuminated by a 
source of white light 3, which is filtered by narrow passband 
filters 4. The filtered light is preferably conveyed to the skin 
2 through a fiberoptic illuminator 5. The light re-emitted by 
the illuminated skin is imaged by a low-noise, high- 
resolution monochrome camera 6, which is preferably an 
electronic charge-coupled ("CCD") camera. Digital images 
output by the camera 6 are provided to a computer 12 for 
processing. 

The computer 12 includes a digital interface 12a, a 
memory 126 and a digital processor 12c. A display 19 is 
preferably provided as well. The computer 12 includes an 
input to a digital interface 12a for receiving the digital 
images. A memory 126 stores the digital images, and the 
software controlling operation of the imaging system, the 
image processing, and the classification and characterization 
of the lesion. The digital processor 12c, under control of the 
software, performs the calculations. The computer 12 has an 
output connected to a display 19, which can display the 
processed images and the results of the classification and 
characterization procedures for each image. The computer 
12 also preferably has outputs connected to the source of 
light 3 and the camera 6, for controlling their illumination 
level and exposure times, respectively, as described below. 

The image processing, classification or characterization 
and other programs can be implemented on a personal 
computer, using a programming language, such as FOR- 
TRAN or C. The memory 126 which stores the software can 
be any convenient media readable by the computer 12, such 
as the hard drive of the computer, read only memory, random 
access memory with a battery backup, electrically pro- 
grammed ROM, electrically erasable ROM, floppy disc, or 
CD ROM. Other suitable media may be used, as well. 

When the filter bandpasses have minimal overlap, as in 
FIG. 1(6), each monochromatic image will contain spec- 
trally independent information. Such spectral separation is 
believed to be useful for differential diagnosis of skin lesions 
that contain varying amounts of melanin, and of hemoglobin 
in different oxidation states, for example. Spectral separation 
is also believed to be useful in distinguishing granulation of 
tissue and other structural details of wounds in various 
stages of healing. One or more of the wavelength bands may 
lie outside the visible region, such as in the near infrared 
and/or the near ultraviolet, as long as the wavelength is 
within the response range of the combined optical system 
including the electronic camera 6. 

In accordance with another aspect of the invention, the 
digital images of skin lesions can be derived from color 
slides of the lesions obtained by clinical imaging, 
dermoscopy, or polarization imaging. FIG. 1(c) is a sche- 
matic illustration of alternative approaches to the acquisition 
and digitization of images of skin lesions from color slides. 
A photo camera 13 produces 35-mm color slides of a region 
of the skin 14. The camera 13 can be a Dermaphot® camera 
from Heine, Optotechnik Grabh & Co. AG, Germany, for 
example. The slides are typically stored in an archive 15. 
The slides are subsequently reimaged by a monochrome 
camera 16, which may be a CCD camera, that photographs 
each slide as it is illuminated by white light that has passed 
through a sequence of bandpass filters 17 to create a color 
filtered version of the image. The slides can be illuminated 
at broad or narrow blue (B), green (G) and red (R) wave- 
length bands, respectively. The broad wavelength bands may 
overlap somewhat. In one example, the blue wavelength 
band was about 400 nm±30 nm, the green wavelength band 
was about 550 nm±30 nm, and the red wavelength band was 
about 700 nm±30 nm. 
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Each of the filtered representations is recorded by the 
monochrome camera 16, which provides the resulting digital 
images 18 to an input of the computer 12. If a CCD camera 
is not used, the slide images could be digitized by any 
available commercial digitizer including three channels, one 5 
for red, one for green and one for blue, as long as the pixel 
size in the lesion plane after digitization is less than about 60 
micrometers ('Tan"). 

An appropriate CCD camera 16 is available from 
Electrim, Inc., Princeton, N.J. The camera 16 has a photo- 10 
graphic macro-lens, wherein f#/2.8 and f«100 mm. 
Preferably, the spatial resolution of the CCD camera 16 
provides pixels having a size about 10-30 //m in the lesion 
plane. The CCD camera 16 from Electrim, Inc., has 753x488 
pixels. The spatial resolution with such a camera is approxi- 15 
mately 21x24 jum at the lesion plane. Digital images of 
lesions obtained with this imaging system were used to 
classify lesions as malignant or benign, and to characterize 
lesions as invasive or non-invasive, as described further, 
below. The Electrim, Inc., CCD camera has 16 has rectan- 20 
gular pixels. A CCD camera with square pixels would 
simplify the calculating procedures. 

Alternatively, a 3-chip CCD camera 20, indicated in 
phantom in FIG. \c, may be used to reimage the slides of the 
region of interest. The CCD camera 20 provides digitized 25 
images for subsequent analysis by the computer 12. Broad 
bandpass filters, which are part of the CCD camera 20, 
produce a representation of the lesion as a set of three 
narrowband images. The filters are typically in accordance 
with CIE Standard Observer, wherein the bandwidths are 30 
broad. 

FIG. 2 is a schematic illustration of the illumination and 
imaging portions of a preferred computer controlled imaging 
system 22 in accordance with the present invention, for 35 
imaging a region of interest of skin including a lesion. The 
electronic camera 23 may be a 10-bit monochromatic elec- 
tronic CCD camera 23, such as the Xillix Model 1400, 
available from Xillix Technologies Corp., Canada. The 
Xillix camera is equipped with wide band, low distortion 4Q 
foreoptics, such as the XJP 1.9/0501, available from Jos. 
Schneider Werke, Germany, The lower distortion fore optics 
and the camera minimize chromatic aberrations of the 
optical system over the "multispectral" sequence of 
exposures, enabling registration of images with sub-pixel 45 
accuracy, over the entire field of view. 

To ensure repeatability of imaging conditions and to 
minimize required intervention by the operator, it is pre- 
ferred that the system be operated at a preset ftstop. For 
cameras such as the Xillix Model 1400, exposure times are 50 
preferably controlled by the computer 12 through an elec- 
tromechanical shutter that can operate reliably between 
minimum and maximum exposure times \ min and \ max . 

The imaging system provides low-noise, high-resolution 
digital images at high data transfer rates, with low distortion 55 
imaging over the entire range of wavelengths covered by the 
collection of filters. The Xillix camera, discussed above, has 
a resolution at the skin surface of about 20 microns per pixel. 

The CCD camera 23 is preferably contained in a hand- 
held unit, represented schematically as box 24. The illumi- 60 
nator source 25 is a tungsten-halogen lamp whose intensity 
is controlled by a light-stabilized power supply 26 whose 
setting is automatically adjusted by the computer 12. A 150 
watt lamp, such as the Phillips EJA, available from Phillips 
Electronics North America Corporation, N.Y, may be used, 65 
for example. The output of the lamp 25 is white light A 
narrowband filter 27 is provided between the source and an 
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optical fiber 28. A plurality of narrowband filters, each one 
corresponding to a different spectral wavelength band, are 
mounted on a filter wheel 29, Preferred filter bandwidths are 
listed in Table 1, below. The filter wheel 29, which is driven 
by a stepping motor 29a, advances each filter to its proper 
position between the lamp 25 and the optical fiber 28, and 
holds each filter in position for a sufficient period of time. 
The computer 12 controls the motor 29a. More or fewer 
filters may be used. Appropriate lenses 14 are provided 
between the lamp 25 and the filter 27, and between the filter 
27 and the optical fibers 28, as well. One or more fiber 
illuminators are provided for conveying the light from the 
source to the lesion. Two such illuminators 30a, 30b are 
shown in FIG. 2 for simplicity. Although the fiber illumi- 
nator illustrated is a bifurcated pair, a ring illuminator which 
provides more nearly uniform illumination at the skin 
surface, is preferred. An angle of illumination of about 20° 
is also preferred. A Fostec Model A0603 available from 
Fostec, Inc., N.Y, may be used, for example. 

The hand-held portion of the system 24 of FIG. 2, which 
includes the camera 23, may be mounted on a cantilevered 
arm (not shown) that can be locked into position. 

The digital signals making up each of the digital images 
output from the camera 23 are provided to the computer 12. 
The computer 12 conducts image processing procedures on 
the digital images to calibrate the images, and to objectively 
segment, estimate parameters, and classify the lesions based 
on the estimated parameters. Operator judgment is not 
required at any point in the process. 

Control is maintained by the computer 12 over source 
intensity, filter position, and such camera settings as shutter 
timing, through the digital interface 12a. Key control param- 
eters are empirically chosen on the basis of feedback from 
histograms of trial images. The intensity of the lamp 25 may 
be maintained at a stable value, commensurate with the 
10-bit dynamic range of the camera 26, by monitoring a 
secondary light source, connected electrically in series with 
the primary light source 25. The light output from the 
secondary source is monitored by a light sensor that is 
optically isolated from light reflections associated with the 
primary source. Such reflections may be caused by the filters 
that are located on the filter wheel, or from the housing of 
the primary light source. This method provides optical 
feedback which is sensitive to changes in light intensity 
caused by changes in lamp lead resistance, for example, 
while it is insensitive to the variable amounts of light 
reflected from the filters, for example. By means of a closed 
control loop, the optical feedback from the secondary source 
is used to maintain long-term constant light output from the 
primary source. 

The apparatus of FIG. 2 can be used for either clinical 
imaging of the skin, wherein the skin is imaged directly, 
dermoscopic imaging, wherein a layer of oil is provided over 
the skin and a layer of glass placed over the oil layer, or 
polarized imaging, where a polarizer 31 is added to mini- 
mize specular reflection as shown in FIG. 2. In dermoscopic 
imaging, the index-matching oil sufficiently reduces the 
specular reflection to avoid the need for a polarizer. 

Instead of being positioned between the light source 25 
and the optical fiber 28, the narrow bandpass filters 27 may 
be placed between the skin and the CCD camera 23 to filter 
the light reflected, scattered and radiated from the skin 2. 

The front end of the system preferably consists of a flat 
glass plate (not shown) for being placed over the skin. Light 
pressure is applied through the glass, onto the skin, through- 
out the imaging process. This helps to stabilize the region of 
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interest against unwanted motion which could blur an image which is more than double the optimal 107 ms exposure time 

or which could lead to spatial misregistration between for light skin. On the other hand; in the near infrared channel 

images obtained in different filter bandpasses. centered at 780 nm, the exposure times listed are much 

The preferred filters 27 for lesion imaging with a shorter, between 24 and 35 ras, and vary relatively little with 

tungsten-halogen white light source 25 have the center 5 skin type. The optimal exposure time for dark skin in the 

wavelengths X ( . and bandwidths (FWHM) listed in Table 1, deep blue channel at 430 nm is at t max =55Q ms, due to the 

for i-1,2, . . . , M, M-10, wherein the bands are labeled by low skin reflectance and relatively low optical throughput of 

j=i-l=0,l, . . . M-l. Such filters are available, for example, tne system at this short wavelength. Even with an exposure 

from Intor, Inc., Tucson, Ariz. In each band, the exposure time this lc >ng, therefore, the image is less than fully 

time is preferably selected to avoid saturation of the detector io exposed. Greater throughput at this wavelength could be 

elements of the CCD camera 23, as well as to maximize the achieved, at the expense of poorer response in the infrared, 

linear dynamic range over which the image data are In Ta °l e the FWHM at 450 nm, is 100 nm, which is 

recorded. These exposure times should be constrained to be mucn broader than for other wavelengths. It has been found 

within limits i min and i max which are related to the electro- that where images are desired for visual analysis as well as 

mechanical design of the shutter, optical throughput of the is computer processing, the broad wavelength band at 450 nm 

camera 23, and avoidance of image blur associated with more closely matches the blue response of the human eye 

motion during the exposure sequence. Suitable values of i min and is therefore preferred. In addition, the broad wavelength 

and i max could be 10 ms and 550 ms, respectively, for hand provides data at a higher signal-to-noise ratio, 

example. The choice of center wavelength and FWHM for Table 1 appears below: 

the filter channels, as well as the corresponding exposure 20 
times, should preferably also take into account the following 
considerations: 

(a) The center wavelength and FWHM for at least two 
channels should be chosen so that characteristic absorp- 
tion lines can be differentiated, such as those associated 25 
with melanin and hemoglobin; 

(b) For a given set of center wavelengths, there are upper 
limits on the associated bandwidths if spectral inde- 
pendence of data in different channels is to be 3Q 
maintained, as illustrated in FIG. 1(a); 

(c) Bandpasses should be chosen in the red, green and 
blue portions of the spectrum which enable "true-color" 
reconstruction of skin images that are suitable for 
visualization by clinicians; 35 

(d) The need for high signal-to-noise ratio in each image 
sets practical lower limits on the product of exposure Tables similar to Table 1 can be readily constructed based 
time and filter bandwidth, especially at short 0 n experimental results for other applications, where other 
wavelengths, where detector response falls off and spectral bands may be better suited. For example, in the 
lesion reflectance is low; and 40 analysis of wound healing, the ability to distinguish oxy- 

(e) The total time taken to acquire the images in all filter genated from deoxygenated blood would be desirable. In 
bands is preferably less than about three minutes, to addition, the wavelengths and exposure times in Table 1 
minimize patient discomfort and possible motion. reflect a balance between the best results for subsequent 

Based on considerations (d) and (e) above, and also taking analysis of the images by a computer, and the best results for 

into account the varying spectral reflectances of skin of 45 visual observation of the images. If visual observation is not 

different colors, the exposure times in each filter channel are necessary, other wavelength bands and exposure times may 

preferably adjustable, with settings based on the dynamic be used. 

range achieved on an empirical basis, with trial images. In FIG. 3(a) describes how the systems and methods of the 

this manner, both dynamic range and signal-to-noise ratio present invention provide for calibration of the recorded 

can be maximized for each filter channel The preferred 50 images. The calibration procedure permits 10-bit image data 

method is to choose t ■ by iteration, based on intensity to be recorded over a large linear dynamic range in each 

histograms of images of the skin obtained with trial expo- spectral band, independent of skin type. The recorded 

sures at each wavelength band. The histograms are analyzed images can also be standardized for diffuse spectral reflec- 

to determine the number of pixels at the saturation intensity tance. Consistent measures of reflectance ratios in different 

level, \ sa =2 b -l (1023 for b=10 bits). The exposure time is 55 spectral bands can therefore be obtained, despite variations 

decreased if the number of saturated pixels exceeds a in illumination pattern with wavelength, changes in the 

predetermined amount, such as 0.01% of the total. position of the illuminator, or aging of the lamp, for 

Conversely, to maintain high signal-to-noise ratio, the expo- example. 

sure time is increased if a predetermined percentile in the First, the effects of dark current and "fixed pattern noise" 

histogram, 99.9%, for example, is reached at less than a 60 are removed in Step 1. N images are recorded by the camera 

preset threshold, such as 99.5% of \ sar The iteration process without illumination. Preferably 8 such dark images are 

typically converges after two or three trials. recorded. The average of these N dark images, ID is calcu- 

The preferred exposure times at each wavelength for lated and stored in the computer 12. 

imaging skin of different colors to classify melanomas are Second, spatial inhomogeneities in the illumination and in 

listed in Table 1, for the embodiment of FIG. 2 with 10 65 the response associated with each CCD pixel are removed in 

filters. It has been found that for the blue channel centered Step 2. A sequence of N* images of an illuminated flat, 

at 450 nm, the optimal exposure time for dark skin is 273 ms, diffuse reflectance standard, such as a white Spectralon® 
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target (R>99%) recorded. As above, N' is preferably 8. The 
N' images are recorded at each wavelength band. To average 
over local inhomogeneities in the reflectance standard, the 
target is moved continuously during the integration time and 
between exposures. A small motor, such as a reciprocating 5 
motor, may be used. The integration time and/or lamp 
intensity are adjusted by the computer 12 at each wavelength 
band until negligibly few of the pixels are at or just below 
an intensity level corresponding to saturation. These N' 
"flat-field" images are averaged to reduce the effect of 10 
spatial non-uniformities in the reflectance standard, as well 
as to improve the detection signal-to-noise ratio. The result- 
ing averages are stored in the computer as \ wi , where i=l, 
2, . . . M. 

Next, monochromatic "raw data" images of the skin, l si9 15 
are captured by the camera and digitally acquired by the 
computer 12 within each filter passband, i=l,2, ... M. If 
dermoscopic imaging is used, where a thin layer of mineral 

011 is spread between the skin and a cover glass is fixed in 
position in front of the camera, each image of the skin 20 
preferably contains an image of a narrow strip of oil-free, 
diffusely reflecting gray material, held in place on the inside 
surface of the cover glass, and located along one edge of the 
field of view. The material may be cut out of a Kodak "18% 
gray" card. Dermoscopic imaging is preferred for melano- 25 
cytic lesions. The alternative clinical imaging mode is 
preferred for the imaging of wounds and bums because 
contact with the wound or burn by a cover glass is not 
desired. Although FIG. 2 indicates a lesion present on the 
skin 2, it will be readily understood that the same method 30 
will apply when a wound or burn is present, instead. In the 
clinical imaging mode, it is preferable to reduce specular 
reflections by employing the polarizer 31, as indicated in 
FIG. 2. 

In either the dermoscopic or clinical imaging techniques, 35 
a fourth step is preferably provided, in which the raw data 
is compensated for dark current and fixed pattern noise and 
then normalized to the flat-field images. The dark-field 
compensation is performed by subtracting the stored average 
dark image 1^ both from the flat-field image and from the 40 
raw data image l si . The ratio of the results of these subtrac- 
tions is then taken. This standardizes the dark-corrected raw 
data to the flat-field image, compensating for spatially 
varying illumination and pixel-to-pixel response variations. 
After the ratio is taken, the result is standardized to the 45 
maximum level, 2 b -l which equals 1023 where b=10 in a 
10-bit data representation. The normalization process thus 
converts the image of the skin and the gray strip into a 
standardized diffuse reflectance map, with the result pre- 
serving a large linear recording dynamic range. In FIG. 3(a), 50 
the dark-field corrected and flat- fie Id -normalized images, 
also referred to as "flat -field-calibrated" images, are denoted 
as l si . In any image, standardization to maximum level can 
be reinterpreted directly in terms of equivalent diffuse 
reflectance on the basis of the average gray level over the 55 
image of the gray strip, <Igray strip> ( . and the measured 
average diffuse reflectance of the gray strip, which is 
approximately 0.2 and varies in a known and repeatable 
manner with wavelength. 

Preferably, the average image intensity in the gray-strip 60 
region is also used to calculate weighting factors for com- 
bining three or more monochromatic images to provide 
"true-color" visualizations of lesion images on the computer 

12 and display 19. This is preferably accomplished in Step 

5, where the user selects the spectral bands to be used in the 65 
color visualization. Step 5 can take place prior to the 
imaging session. Four bands are currently preferred for such 
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visualization. Filter bands j-3 and 8, in a 3:2 ratio, for the 
red (R) channel, filter band j=2 for the green (G) channel, 
and filter band j=9 for the blue (B) channel, in Table 1. As 
indicated in Step 6 of FIG. 3(a), the relative weights applied 
to the R:G:B channels are preferably inversely proportional 
to <Igray strip>, the average intensity over the portion 
occupied by the gray strip area in each image. This proce- 
dure tends to reconstruct the hues and saturations in the 
original scene to within accuracy limits associated with 
response nonlinearities of the display 19. To minimize the 
effects of such nonlinearities with display monitors such as 
the Sony Model GDM-175E1 Multiscan monitor, for 
example, the viewer may prefer to adjust the maximum 
brightness in the image to correspond to the maximum 
image intensity level of the monitor. A linear transformation 
step, which can be readily accomplished by commercial 
software such as Adobe Photoshop, may be used. If the 
digital images are derived from photographic slides, as in the 
embodiment of FIG. 1(c), steps 5 and 6 are not necessary. 

As indicated by dashed lines in FIG. 3(a), either the 
normalized monochromatic images resulting from Step 4 or 
the color visualization provided from Step 6 can be dis- 
played on the display 19. Any or all of the monochromatic 
raw images could be displayed as well. 

FIG. 3(/>) is a flow chart of a preferred method of 
processing images according to the present invention for 
characterizing the condition of a region of interest of the skin 
of a subject which includes a skin lesion. A skin lesion is 
selected at Step 50. Digital images of the lesions illuminated 
by light filtered at the desired wavelengths of X 2 -X 4 . . . , are 
digitally recorded in Steps 52, 54, 56 and 57. . . , as described 
above. Each of these digital images is processed separately. 
In Step 58, the image taken at the lowest wavelength band 
is used to create a mask for segmentation. At Steps 60, 62, 
64, 65 ... , each of the images of the lesion that correspond 
to different wavelengths are segmented by means of the 
segmentation mask obtained at Step 58. Lesion parameters 
are computed from each of the segmented images, in Step 
66. Lesion parameters found to be useful for classifying and 
characterizing the lesion and statistical methods for com- 
puting the estimated values of the parameters, are discussed 
further, below. The estimated values of the parameters are 
provided to a linear classifier in Step 68. The linear classifier 
employs a linearly weighted sum of the individual param- 
eters to derive a value used to classify the lesion as malig- 
nant or benign. A non-linear classifier such as a Gaussian 
quadratic classifier or an artificial neural-net classifier, each 
employing a suitable defined merit function, may be used as 
well. In either case, the numerical value produced by the 
classifier is subjected to a threshold test at Step 100, such 
that if the test is passed, the lesion is suspected to be 
malignant melanoma. If the test is failed, the lesion is 
declared not to be melanoma. The lesion could also be 
characterized as invasive or non-invasive with a different 
classifier. 

I. SEGMENTATION 

The segmentation algorithms will now be described. The 
function of the segmentation algorithms is to discriminate 
between the lesion and normal skin in the field-of-view of 
the imaging device. This is a complex function since not 
only is the lesion appearance highly variable but so is the 
appearance of healthy skin due, for example, to the presence 
of blotches, hair, wrinkles, etc. The automatic algorithm 
described here is based on the images in the blue spectral 
band, from about 400 nanometers (nm) to 500 nm. This 
spectral band was selected because melanin absorption 
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increases rapidly as the wavelength decreases. While the use respectively. FIGS. 1(b) and 1(e) are images resulting from 

of ultraviolet radiation could be advantageous, since ultra- the iterative thresholding described above. Various dark 

violet radiation is carcinogenic, only low doses can be used. blobs are seen in the images outside of the lesion area. These 

Segmentation in blue consists of several automatic steps: are removed in the following step. 

Location of major peaks in the histogram 5 Image cleaning 

First, the histogram of intensity levels in the whole image Some of the blobs in the thr esholded images arise natu- 

is determined. Then given a sliding window with the range faU due eithef to dafk Qn me normal skin of tQ haif 

of (2L+1) intensity levels, the number of peaks N D in the . „ T „ ~ . t , . „. , 

. tU t • » . • j if kt i .if as in MG. 7(d). Others are artiracts such as the film edge at 

histogram over that range is determined. It N B <2, the range in , P . . — v . . . , . 

is decreased by two levels and if N p >3, the range is increased 10 the t0 P of lhe nevus ima S e in FIG * ^ or dark bands al the 

by two levels and the process is repeated until N p «2 or 3. For ima S e ed 8 es from lne sllde mounts - These bands are 

most of the images in the data base used in this study, there removed by automatically testing for their presence and then 

are two major peaks in the histogram. Examples of such setting the intensity of appropriate pixels to zero. The 

histograms are shown in FIG. 4(a) for a malignant mela- remaining blobs could also be removed by determining the 

noma and in FIG. 4(b) for an atypical melanocytic nevus. overall number and size, i.e., number of pixels, of connected 

The lesions correspond to the lower intensity peak, since it blobs, and then setting to zero the intensity of pixels 

is darker than the surrounding skin due to strong absorption belonging to the small ones. However, since the size of some 

by melanin at 400 nm. However, some lesions are quite lesions exceeds 10Q QO0 pixels, this would be computation- 

inhomogeneous, and the automatic procedure described can nn „ ... ™ c . . ... 4 

a a i ■ i *ii . * a - n/-c =/ \ a c/*a 20 a Hy vei 7 intensive. Therefore, in practice, this step is 

find 3 major peaks, as illustrated in FIGS. 5(a) and 5(b). ^ , , ■ , r .. . <* 

... , . 1 1 preterably carried out as follows. First, perimeter pixels for 

Location or the intensity threshold „ , , , . . . . , ™_ , r ... 

J all blobs in the image are located. The number of such pixels 

If two major peaks are found in the intensity histogram, is icaU , ess , han 10 000 ^ each of these imeler 

then the threshold value \ th is selected to be at the histogram . , . , . . L1 , , , 

lh . ..... pixels is assigned to a unique blob and its size, the number 

minimum between these two peaks, as indicated in FIGS. 25 r f . . . , . ,7 , • a * * a t-l • * 

At \ a a/u\ i *u cZ i •„ u u e a of perimeter pixels in the blob, is determined. The intensities 

4(a) and 4(b). In the case of three peaks, it has been found * • , . ; , , , c . i . r . 

that, if the middle peak is closer to the lowest intensity peak, of P lxels belon f n & 10 blobs of slze less lhan 30% of the 

the threshold value is at the minimum between the middle maximum size for that image are set to zero. This process is 

and the highest intensity peak. If the middle peak is closer iterated unlil a11 the sma11 blobs are amoved. Typically less 

to the highest intensity peak, then the threshold value is at 30 than 10 iterations are needed. The intensity of all the nonzero 

the minimum between the middle and the lowest intensity P ixels & tnen set to 1. The resulting binary lesion mask has 

peak, as shown in FIGS. 5(a) and 5(b). the following property: 
Iterative thresholding of the image 

The next step in image segmentation is iterative thresh- i B ( x , jy) = { lf pI * el 31 ^ belongs t0 lesi0n; ( 4 ) 

olding of the images. Given the intensity threshold value, 35 l°* 0thenvis& 
image thresholding has been typically accomplished as 

follows. The intensity I(x,y) of a pixel at location (x,y) is set ™™ „x , -n . , . 

to zero if it exceeds l rt> i e , F,GS 7c > and 7 W llluslrale the ' es ™ 

In the images illustrated in FIGS. 1(a) and l(d) y dark hairs 

(i(x,y), if i{x, y)<i, h \ 0) 40 were either absent or were not adjacent to the lesion. 

= 1 0 otherwise. However, there are many images with prominent dark hair 

overlapping lesions. Segmentation of such images is 
described in the following section. 

FIGS. 6(a) and 6(d) are examples of digital images of c„™ 0 „«.«^„ ~p • • „ MOOrt „ a nf 

v 7 , - . i . b . ... Segmentation ot images in presence or hair 

malignant melanoma and atypical melanocytic nevus in the 45 ^ 

blue spectral band, respectively. FIGS. 6(b) and 6(e) are FIGS - 8 W and 8 W are examples of lesion images with 

images resulting from the direct thresholding as in Eq. (1). hair - since the segmentation algorithm described in the 

As shown in FIGS. 6(b) and 6(e), "holes" can appear within previous section would leave some of these dark hairs 

the lesion. Therefore, an iterative approach is preferably connected to the lesion, images with hair require special 

used. First, the intensity of pixels at the image edges is set so preprocessing to allow for hair removal from the normal 

to zero. Then as each iteration proceeds, the intensity I(x,y) skin. Since hair is a problem because of its high contrast with 

of a pixel at location (x,y) is set to zero if it exceeds l th and respect to the normal skin in the blue, a spatial filter was 

at least one of its nearest neighbors has zero intensity, i.e., designed to locate hairs. This filter, shown in FIG. 9, is 

magnification dependent. It is applied to every pixel of the 

/ (x y) - l °* if ^ * l,h Md Nm = °' (2) 55 °"S ma l image and the result is thresholded at the 5% of 

1 ~\ i(x, y), otherwise, maximum value in the whole filtered image. The filtered 

images are shown in FIGS. 8(b) and 8(/) in reverse intensity 
contrast, wherein bright features are dark. Hairs are clearly 
located in the filtered images. It should be noted that the 

N nn -m\ili(?-iyy(x+ic>)jfcy^i)j(&y+i),]. (3) lesion interior is almost entirely blank, indicating poor 

™ . - • , ... contrast between hair and lesion. 
This procedure is iterated until there are no pixels with 

I(x,y)£I rt> and a nearest neighbor with zero intensity. Hairs are removed by an averaging process. For every 

Typically, only a few iterations are required to complete this non-zero pixel at (x,y) in the filtered image one finds the 

step. The resulting images are shown in FIGS. 6(c) and 6(f). 65 locations of 4 nearest pixels (x 7 , y), (x w ,y), (x, y,), (x,y M ) 

FIGS. 1(a) and 1(d) are other examples of digital images (where x,<x<x M and y,<y<y M ) with zero intensity. Then the 

of malignant melanoma and atypical melanocytic nevus, intensity of every pixel in the original image that has 



where 
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non-zero intensity in the filtered image is replaced as fol- monochrome camera 16 was used to digitize slides. The 
lows: subscript r, g, or b refers to the red, green, or blue spectral 

band in which the parameter is evaluated. If additional 
y) = < 5 ) spectral bands are used, then each of the parameters could be 

5 computed at the additional spectral bands, as well. 

1 ^ 

— ^ [/(jc„ +k,y) + i(x t - k y y) + /(.v, y u + k)+ t(x, y t -k)]. Specific algorithms for these parameters are described 

*=i below. For simplicity it is assumed that the image pixels are 

square but the algorithms described below may be imple- 

The images averaged in this way are shown in FIGS. 8(c) 10 menled for rectan S ular P ixels as weIL 
and 8(g). It is seen that the contrast between hairs and Lesion Asymmetry 

normal skin is considerably reduced in these images. After 

this preprocessing, the segmentation algorithm described in Asymmetry parameter 

the previous section is applied to the averaged image. The The lesion asymmetry parameter is based on moments of 

final binary lesion masks are shown in FIGS. 8(d) and 8(h). 15 tne intensity distribution. First, the lesion orientation angle 
The preprocessing step described above may be used for e i s use d to locate the principal axes, which are just the 
all lesion images, regardless of the presence of hair, enabling symmetry axes for symmetric lesions. The angle 6 is corn- 
fully automated lesion segmentation. However, since this puted from 
requires more computation and causes some border blurring, 

the need for preprocessing due to the presence of dark hair 20 2({x - x c ){y - y c )) (7) 

is preferably indicated interactively by an operator, and tan 20 = ^ ^ _ ^ _ , 

images preprocessed only when necessary. 

Segmentation of images in other spectral bands 

Since melanin absorption is strongest in the shortest- where the lesion intensity centroid is at 
wavelength band, the lesion area, which appears as a dark 25 

region in the image, appears largest in the blue spectral band. v<*> and y c =<y>- (8) 

Since longer wavelength radiation penetrates deeper into ^ angular brackets in £ {J) and (g) denole an imensi 
skin, if the thickness of the melanin-containing layer com- momentj which fof any fonclion f(x y) of posilk)n in the 
pensates for the weak absorption, that part of the lesion will image can be computed as follows: 
appear dark even in the red spectral band. For thick 30 

melanomas, with Breslow thickness greater than 1 mm, one ^ ^ 

expects dark lesions even m the infrared bands. This was 2j Zj y ^ L ^ ^ 

observed, for example, by Marchesini et al., Photochemistry </(*, y)) = 

& Photobiology, "In vivo spectrophotometric evaluation of 
neoplastic and non-neoplastic skin pigmented lesions. III. 35 
CCD camera-based reflectance imaging," Vol. 62, 1995, pp. 

151-154. However, for early malignant melanomas, with where W x - 15 the segmented lesion image. In order to 
Breslow thickness less than 1 mm, great variability of compare properties of different lesions, the parameters used 
images in the red spectral band has been found. There may ™ independent of the orientation of the lesion in the image, 
be so little contrast between the lesion and the normal skin 4n Therefore, the lesion asymmetry is determined with respect 
that direct segmentation is not possible. Therefore, segmen- 10 the P nncl P al axes - ^ measure of asymmetry described 
tation of lesion images in all spectral bands with wavelength here rec l uires rotatlon of the ima S e bv an an S le 0 50 that 
X uses the binary lesion mask of Eq. (4), obtained in the P"ncipal axes are parallel to the image axes. In this 
shortest-wavelength band, here blue, i.e., principal-axis coordinate system the following asymmetry 

factors are defined: 

(6) 45 

FIGS. W(a)-W(f) are a series of images of the lesions, Z Z 1 1l{x < + & ~ " «• ^ < 10a > 

with their corresponding histograms shown in FIGS. 5(a) A x = ZZ/Jjc >) * 

and 5(b), segmented in the blue, green, and red spectral * y L 

bands, as indicated. The automatically determined lesion 50 
borders are superimposed on the original lesion images. The 



LZki*. y) 



area of dark regions is largest in the blue. 

II. LESION PARAMETER ESTIMATION 

Objective and automatic lesion classification requires 55 

quantitative algorithms for lesion parameter estimation from ^ as y mmetr y parameter, 

their segmented images. Such parameters should be AtsA +A 

dimensionless, independent of lesion location and orienta- * y ' 

tion in the image, and of the overall image brightness. It is is a measure of asymmetry in the geometric shape of a lesion 

convenient to separate the parameters used here into four 60 as well as in the distribution of lesion pigmentation. Asym- 

broad classes: asymmetry, blotchiness, border, and texture. metry parameters tend to be larger for malignant melanomas 

Parameters with the highest diagnostic accuracy for malig- than for atypical melanocytic nevi. 

nant melanoma are listed in FIG. 11, together with the values Binary asymmetry parameter 

of diagnostic accuracy, sensitivity, and specificity, for a If the intensity distribution l L in Eqs. (10a) and (10b) is 

training set of images of 41 malignant melanomas and 104 65 replaced by the binary intensity distribution of Eq. (4), then 

atypical melanocytic nevi obtained with the imaging system the corresponding asymmetry parameter A bin is the fraction 

described above, with respect to FIG. 1(a) wherein the of the lesion pixels which do not have a counterpart on the 
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other side of the principal axis. Thus, when based on the channels are equal for white. In the spherical color coordi- 
binary intensity distribution, parameter A^ is a measure of n ate system, 
the asymmetry of the geometric shape of the lesion. 

R hix, y) (16) 

Lesion Blotchiness 5 ~ /*(*, y) + i B {x, y) + / c u, y)* 

Visually, many early malignant melanomas appear C(x, y) = ^ ^ + ^ y ^ + ^ ^ , 
blotchy. In multispectral images there may be darker and 
lighter regions or blotches of rather homogeneous intensity. 

In color images, in contrast, there may be regions of different 10 where the subs cripts R, G, B refer to red, green, and blue 

colors. Therefore, it is of interest to quantify such blolchi- a , bands> are chosen ^ the ind denl variables . ^ 

ness in order to differentiate malignant from benign lesions. , . . . , A . . . . - n „. 

& 6 lesion is then divided into color regions as follows. First 

Blotchiness Parameter Based on Spatial Intensity Distri- R( x , y ) and G(x,y) are divided into and N a topographic 

DUt i° n regions. A color region is defined as a particular combination 

The lesion is divided into N, "topographic" regions. If of two topographic regions. The total number of color 

I m/]¥ and \ min are the maximum and minimum intensities in regions is 

max min c 

the lesion in some spectral band, respectively, then a pixel at 

(x,y) belongs to the nth region if n<*Nr*Ng- O 7 ) 

20 

, /m« - /min , W - 'min (12) The blotchiness parameter based on color is defined in 

/mii> + (n-l) Tj 5 l L (X, y) < /min + tl . r 

N t analogy with Eq. (14): 

n c (18) 

For nth topographic region defined in Eq. (12), a distribution y Vvar(4J 

of distances of pixels in that region from the intensity 25 _ «=i 

centroid of the binary lesion mask " N c 



d n (x, y) = V {x„ - x c ) 2 + (y n - y c ) 2 



(13) 



30 



Lesion Border 



is obtained and its mean value <d„> and variance Var(d„) are 

computed. The measure of lesion blotchiness based on Border Irregularity Parameter 

spatial intensity distribution is 

35 Border irregularity is a well-known feature of malignant 

(14) melanomas. It is typically defined as the ratio of the mea- 

] Vvar(4) sured lesion perimeter to the perimeter of a circle with the 

bi=^L- , same area as the lesion. Since perimeter is difficult to 

£ (d n ) estimate reliably, a statistical descriptor of border irregular- 

" =l 40 ity is used here. In addition, many lesions are elongated and 

an ellipse is a better approximation for such lesions with 

This parameter can be evaluated in every spectral band. regular borders than a circle. 

Blotchiness Parameter Baled on Centroids Using lhe binary lesion mask of Eq ( 4)) the lesion 

The lesion is again divided into N, "topographic" regions 45 intensity centroid from Eq. (8), orientation angle from Eq. 

as defined in Eq. (12). An intensity centroid (x c (n),y c (n)), (7^ areaj and tne aspect ratio defined as 
defined in Eqs. (8) and (9), is then computed for each such 

region separately. The blotchiness parameter based on the < (19) 

centroid is defined as AR _ ~*c) 



C-{X max -X min ){Y max -Y min )IA l (15) 



50 



V</-*> 2 



where, for example, X max is the maximum value of x c (n), where fefef tQ the ^0^^ system defined by the 

and A, is the lesion area in pixels. This blotchiness parameter ^ incj j axes> afe determined ^ values are then 

is also determined in each spectral band separately. ^ ^ 1q ^ ^ . g ^ b ^ regulaf apprQxi _ 

Blotchiness Parameter Based on Spatial Color Distribu- mation tQ the Jesion border For each lesion bofder pixe , at 

tlon ( x z,»yfc)» i^ angle with respect to the horizontal axis: 

The "color" in this analysis is not related to the visual 

perception of color. It is a quantitative descriptor of the {x b ~x c ) (20) 

relative intensities in red, blue, and green channels in a 60 0 = tan~ ^ _ ^ , 
particular pixel. 

All the other lesion parameters described here involve 

analysis of images in each spectral band separately. and the location of the ellipse border for the same angle 

Therefore, absolute calibration of image intensities was not WWy^)) are determined. The distribution of distances 

necessary. However, in order to describe the color 65 between the ellipse border and lesion border: 
distribution, normalization of intensities in red, green, and 

blue spectral bands is needed, so that intensities in the three ^(wKlwt)-^ (21) 
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where 



20 

-continued 



d b (x b , y b ) = yj (x b - x c ) 2 + {y b - y c ) 2 
and 



(22) 



and 



72 = VVar(C, ) 



(29) 



4(0) = ^x 2 , + yh 



(23) Another texture parameter uses the following variable: 



is obtained and the border irregularity parameter is defined 1° 
as 



C3(^)=4KO ) O)+Vv(-/I ( ,0>VV(rt,,0)+H<0,-rt>H'(0^J-2[H<-«„-«> 

w (-n,/t t )+w(n lt -n^+w(n lt «,)] (30) 



where 



War(^) 
<d b ) 



(24) 



(31) 
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Border Gradient Parameter 

Another parameter that quantitatively characterizes lesion 
border is the measure of intensity gradients across the lesion 
borders over the length scale defined by n g , in units of pixels. 20 
For each lesion border pixel at (x fo ,y fc ) one determines 
whether pixels at (x fr ±n 5 ,y fc ±n^) are at the border. If they are 
not, then the gradient is defined as 

G(x^=^[l/(x+/i fi ,>)-/^-/ 1 ^)|+l/(^>'+« x H(^>'-» s )|]^ (25a) 25 

otherwise, if pixels at (x±n^,y) are not on the border, 



If the value of C 3 is negative, it is set to zero and the 
corresponding texture parameter is 



73 = 



VVartC 3 ) 
<C 3 ) 



(32) 



Another variable that leads to a texture parameter useful for 
classification of melanocytic lesions is: 

^(^-SH^O^VK-ZI^O^VV^^O)-^-^)-^/!^-^^/!,,-/!,)- 

^-",A)-»<«n->0-"'(",A) (33) 



G(x b j> b )»\l(x+n gt y)-I{x-n g> y% 
or, if pixels at (x,y±n g ) are not on the border, 

G(x biyb )=\l(x,y+n g )-I{x,y-n g )\. 

The border gradient parameter is defined as 



(25b) Again, if the value of the variable is negative it is set to zero 
30 and the corresponding texture parameter is 



(25c) 



T4 = 



VVar(C 4 ) 
<C 4 > ' 



(34) 
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VVar<C) 



(26) 
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Lesion Texture 

The description of lesion texture is particularly vulnerable 
to subjective judgement. The quantitative evaluation of 
lesion texture parameters is possible only using computer- 
based image analysis. While many such parameters are 
possible, those found to be helpful in discriminating 
between malignant melanomas and atypical melanocytic 
nevi are described below. 

Texture Parameters Based on Local Intensity Variations 

Texture parameters are defined over a length scale n, in 
units of pixels. For example, consider a pixel located at (x,y) 
in the lesion. Let I, and \ u be the minimum and the maximum 
intensities in an image in the 2n,+lx2n,+l window around 
this pixel, i.e., in the range [x-n„ x+n,] and [y-n„ y+n,]. 
Consider a variable 



45 
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Texture Parameters Based on Pigmented Network 
Texture parameters have also been developed by consid- 
ering the properties of a pigmented network. These texture 
parameters are measures of variability in the area of the 
dermal papillae and in the aspect ratio (length/width) of the 
rete ridges. 

Since dermal papillae appear as the brighter part of the 
network, one seeks all the local maxima over a 2n,+lx2n,+l 
window. Starting from such a maximum at (x m ,y m ), one 
finds local one-dimensional minima in eight directions (2 
vertical, 2 horizontal, and 4 diagonal) and locates the 
vertices of an octagonal region one pixel closer to the 
maximum intensity pixel than the minimum pixel. Such 
octagonal regions approximate the areas of dermal papillae 
A dp which are computed from the known location of verti- 
ces; the corresponding texture parameter is 



55 



T5-. 



VV^V) 



(35) 



C,U, y) 



(27) 



The first two texture parameters are defined as: 



Tl = 



(CO 



(28) 



Some of the areas determined by this algorithm are due to 
bubbles visible in some of these dermoscopic images. 

60 However, since there are typically on the order of hundreds 
of areas, and on the order of tens of bubbles, the statistical 
parameters should not be significantly biased by this artifact. 

The aspect ratio of rete ridges is determined in a similar 
fashion, although one starts with local minima since rete 

65 ridges appear dark in the images. The vertices of an octago- 
nal region are determined in this case from one-dimensional 
maxima in the eight directions. The maximum and minimum 
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extents of this region are then determined and the aspect Uk)>L th , and as benign otherwise. The resulting classifier 

ratio R is computed. This texture parameter then is for the training set is 

/ (36) L 1 =0.Q25A bin +QWQA b +QM9A g +0A6QA r +0328C b +Q.Q95a+ 

T6 = ——^- 0.0385+0.107n s +0.064r2 a +0.018n,+0.11173 i ,+0.l6773 if + 

<*> 0.26875 b (38) 

where the weights are normalized so that the threshold value 

III LESION CLASSIFICATION equals one. This classifier with sensitivity to malignant 

melanoma of 100% and specificity of 85% is shown in FIG. 
Selection of lesion parameters for classification was done 10 12. Statistical significance of the specificity and sensitivity 
by determining the maximum diagnostic accuracy for malig- was assessed by considering the binomial probabilities for 
nant melanoma for each parameter computed in every the value of L 3 to exceed the threshold for the 41 malignant 
spectral band available for the training set of images. As melanomas and 104 atypical melanocytic nevi of the train- 
mentioned above, diagnostic accuracy, sensitivity to malig- ing set separately. At the 95% confidence level, one finds 
nant melanoma and specificity for the selected twenty two 15 that sensitivity is not less than 93% while specificity is not 
parameters are shown in FIG. 11. These parameters were less than 79%. Since there are several melanomas very close 
then used as input to the linear classifier. Nonlinear classi- to the threshold value, a practical classifier may use a 
fiers may be used as well. threshold value that is less than one. It has been found that 

For each lesion k the linear classifier is ,„ this s r el of 14 * im A a 8 es is sufficient to yield statistically 

20 significant results. A greater number of images may be used, 

(37) aS WelL 

Uk)= Y w R x a (k), Some of the noninvasive melanomas, called melanomas 

„=i " ■ in-situ, are confined to epidermis and are 100% curable by 

surgery. The invasive melanomas, i.e., superficial spreading 

. 25 melanomas in our data base, require more extensive surgery, 

where X„(k) are the parameters for the kth lesion and Therefore, it is of clinical interest to differentiate between 

weights w„ are to be determined so that a specified function invasive and noninvasive me lanomas and a linear classifier 

F(L) attains maximum value. The following functions F(L) was tfained tQ perform this task ^ classifier> with weights 

were used: 1) specificity under constraint of 100% sensitiv- chosen tQ maximize the over . all classification accuracy for 

ity to malignant melanoma for the training set which 3 o the 2 4 superficial spreading melanomas and 16 melanomas 

included 41 malignant melanomas and 104 atypical melano- in-situ of the training set is 
cytic nevi; (2) classification accuracy for the 24 invasive and 

16 noninvasive malignant melanomas of the training set; and L 2 —i.0QA b -Q.UBl g -2AiBl r -QAC b -Q.98Cl -iA7T2,+0.53T4 b + 

3) correlation with the Breslow thickness for the 24 invasive i.9875 6 +i.587"6 6 -0.73 (39) 

malignant melanomas. 35 

Given any training set of lesion images and corresponding where a constant was subtracted from the classifier values to 

set of lesion image parameters, the weights that maximize obtain the threshold value of zero. This classifier, with 

F(L) are found as follows. First, an initial range and reso- overa11 classification accuracy of 92.5%, is shown in FIG. 

lution A for w„ are selected. For each allowed set of values ^ 

of w„, the values L(k) are computed for each lesion. The 4 o Since P ro 8 nosis for invasive melanomas correlates 

value F(L) is determined based on the input from histo- strongly with the Breslow thickness, a linear function of 

pathological evaluation of the lesion based on a biopsy, such lesion parameters Q was trained to maximize the Pearson 

as the diagnosis of the lesion as benign or malignant, and the correlation co-efficient between Q and the Breslow thickness 

Breslow thickness for a malignant melanoma The range of for a set 24 superficial spreading melanomas. This function 

w„'s is adjusted until the maximum value of F(L) is inside 45 1S 

the range. Then the resolution A is reduced by a factor of ^ „^r. , ,, m „, ^ 

two, and the process is repeated until A reaches specified 0.5065+0.20271 +1.47677,-0.485 (40) 

minimum value A mi>1 , This procedure determines the weights * 

w n only up to a multiplicative constant. It is noted that the and is shown in FIG. 14. Even though there are only 24 

classifiers resulting from a particular training set are appli- 50 superficial spreading melanomas in the data base, the high 

cable only to images with a specific spatial and spectral correlation of 0.77 is statistically very significant since 

resolution, and that lesion images obtained with a different p=9xl0~ 6 . 

imaging system may require the development of different The classifiers of Eqs. (38)-(40) are applicable to the 

classifiers, by the procedures described above. imaging system described above, with respect to FIG. 1(a) 

Since detection of melanoma in its early stage signifi- 55 wherein the monochrome camera 16 was used to digitize 

cantly improves prognosis, there is a need for reliable slides. 

methods of early detection. Clinical evaluation of melano- For other imaging systems, having different spatial and 

cytic lesions is, however, a problem since reliable differen- spectral resolution, different classifiers may need to be 

tiation between early malignant melanoma with Breslow developed, based on a sufficient data base of lesion images 

thickness less than 1 mm and atypical melanocytic nevus is 60 obtained with that imaging system, in accordance with the 

difficult even for experienced dermatologists. In order to procedures described above. 

detect as many early melanomas as possible, weights in the The segmentation, parameter estimation and classification 

linear classifier are preferably chosen to maximize specific- programs described in Sections I — III, above, can be imple- 

ity under the constraint of 100% sensitivity to malignant mented on any personal computer, using a programming 

melanoma for the training set. For each set of weights, one 65 language, such as FORTRAN or C. The program can be 

finds the threshold value L lh of the linear classifier such that stored on any convenient media readable by a computer, 

a kth lesion is classified as suspicious of malignancy if such as read only memory, ("ROM"), random access 
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memory with a battery backup, electrically programmed 
ROM (EPROM), electrically erasable ROM (EEPROM), 
floppy disc, CD ROM, or a hard disc. Other suitable media 
may be used, as well. 

While the procedures of Sections I — III were described 5 
with respect to digital images obtained by imaging color 
photographic slides of skin lesions with a monochrome CCD 
camera 16 in accordance with the system of FIG. \c, these 
procedures are readily adaptable to the analysis of digital 
images of skin lesions acquired directly from the region of 
interest of the skin with a monochrome CCD camera 6 of 
FIG. la and FIG. 2. 

In the process described in Sections I — III, above, segmen- 
tation was conducted in the blue wavelength band. The 
segmented mask in blue was then applied to images in the 15 
red and green wavelength bands. Where images at additional 
wavelengths are provided, segmentation is preferably first 
attempted at the shortest available spectral band where the 
contrast between the lesion and normal skin tends to be 
highest because of strong absorption by melanin. The seg- 2Q 
mented mask is then applied to the images in the remaining 
spectral bands. These steps are shown in FIG. 3b. 

In addition, where parameters are described in terms of 
the red, green and blue wavelength bands in Section I — III, 
parameters can be derived at each of the other wavelengths 25 
used in the systems and methods above, in accordance with 
the procedures described in Section I— III. The additional 
parameters can be readily used to develop a classifier, also 
by the processes described in Section I— III. 

The references cited above are incorporated by reference, 3Q 
herein. 

While preferred systems and methods for practicing the 
present invention have been described above, it is under- 
stood that departures may be made from the systems and 
methods, without departing from the scope of the present 35 
invention, which is defined by the following claims. 

We claim: 

1. A method of characterizing a skin lesion wherein the 
absorption and scab of light in different spectral bands by the 
skin lesion is a function of the condition of the skin, the 4Q 
method comprising: 

illuminating a portion of the skin including the region of 
interest by light in at least three spectral bands; 

digitally imaging a portion of the skin including the region 
of interest at the at least three spectral bands with the light 45 
re-emitted by the portion of the skin to generate digital 
images comprising digital signals whose values are a 
function of the condition of the region of interest of the 
skin; and 

providing the digital images to a processor, wherein the 5Q 
processor: 

segments the digital image by generating a single seg- 
mentation mask defining the boundary of the region of 
interest for each image, where the single segmentation 
mask is the segmentation mask having largest area of 55 
segmentation masks generated from each image in each 
of the at least three spectral bands, without operator 
intervention; 

automatically computes at least one estimated value for 
each digital image at each spectral band which is a 60 
function of a characteristic of the portion of the region 
of interest determined by the segmentation mask, with- 
out operator intervention; 

characterizes the condition of the skin as malignant or 
benign based on the estimated values, without operator 65 
intervention; and 

outputs the characterization of the condition of the skin. 
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2. The method of claim 1, further comprising estimating 
at least one value which is a function of the texture of the 
region of interest. 

3. The method of claim 2, wherein the computing step 
comprises estimating values which are statistical measures 
of local intensity variation in the digital images in each 
spectral band, which are a function of the texture of the 
region of interest. 

4. The method of claim 2, wherein the computing step 
comprises estimating values based on the ratio of standard 
deviation of the areas of dermal papillae to their mean within 
the segmentation mask. 

5. The method of claim 2, wherein the computing step 
comprises estimating values of the average and standard 
deviation of the thickness of rete ridges within the segmen- 
tation masks. 

6. The method of claim 1, further comprising estimating 
at least one value which is a function of the asymmetry of 
the region of interest in each spectral band, for two principal 
axes of the segmented image by: 

determining the principal axes of the segmented image; 
rotating the principal axes of the segmented image until they 

are oriented parallel to the coordinate axes of the image; 
computing the differences in intensity between each pair of 

pixels whose locations, with respect to a principal axis, 

are mirror images of each other; 
summing the absolute values of said intensity differences; 
calculating asymmetry values with respect to each principal 

axis, by normalizing the said sum to the total intensity in 

the segmented images; and 
adding together the asymmetry values calculated for the two 

principal axes. 

7. The method of claim 1, further comprising estimating 
at least one value which is a function of the blotchiness of 
the region of interest. 

8. The method of claim 1, further comprising estimating 
at least one value which is a function of the irregularity of 
the border of the region of interest by estimating a value 
which is a statistical measure of the deviation of the border 
of the segmentation mask from the border of an ellipse of the 
same area, aspect ratio, and orientation as the segmentation 
mask. 

9. The method of claim 1, further comprising estimating 
a value which is a function of the gradient at the border of 
the region of interest by estimating a statistical measure of 
the gradient values of the intensity of the digital images 
across the border of the segmented images, at each spectral 
band. 

10. The method of claim 1, further comprising character- 
izing the type of lesion as invasive or non-invasive. 

11. The method of claim 1, wherein the segmenting step 
comprises generating the segmentation mask from a digital 
image by: 

removing digital signals from the digital image which 
correspond to hair structures; 

deriving a threshold from a multimodal histogram of 
intensity levels; 

iteratively applying the threshold to the digital signals of 
the digital image; and 

removing digital signals which correspond to small blob- 
like regions from the masked image, 

12. The method of claim 1, wherein the digital imaging 
step further comprises digitally imaging the region of inter- 
est with a digital camera. 

13. The method of claim 1, further comprising: 
photographing the region of interest with a color camera 

to form color photographic slides; and 
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illuminating the color photographic slides with light in 
each spectral band; 

wherein the digital imaging step comprises digitally imag- 
ing the illuminated color photographic slides of the 
region of interest with a digital camera. 5 

14. The method of claim 1, where the characterization 
step is based on a non-linear combination of the estimated 
values. 

15. The method of claim 1, where the characterization 
step is based on a linear combination of the estimated values. 10 

16. The method of claim 1, where the characterization 
step is based on a sequential combination of applying a 
linear combination of the estimated values and a non-linear 
combination of estimated values. 

17. A method of characterizing the condition of a region 15 
of interest of skin, wherein the absorption and scattering of 
light in different spectral bands by the region of interest is a 
function of the condition of the skin, the method comprising: 
illuminating a portion of the skin including the region of 

interest by light in at least thee spectral bands; 20 
digitally imaging the portion of the skin including the region 
of interest at the at least tree spectral bands with the light 
re-emitted by the portion of the skin to generate digital 
images comprising digital signals whose values are a 
function of the condition of the region of interest of the 25 
skin; and 

providing the digital images to a processor, wherein the 
processor: 

segments the digital images by generating a single seg- 
mentation mask defining the boundary of the region of 30 
interest for each image, where the single segmentation 
mask is the segmentation mask having largest area of 
segmentation masks generated from each image in each 
of the at least three spectral bands; 

computes at least one estimated value for each digital 
image at each spectral band which is a function of a 
characteristic of the region of interest determined by 
the segmentation mask; 

characterizes the condition of the region of interest of the 40 
skin based on the estimated values; and 

outputs the characterization of the condition of the region 
of interest of the skin. 

18. The method of claim 17, wherein the estimating and 
characterizing steps are conducted without the intervention 45 
of an operator. 

19. The method of claim 17, wherein the segmenting step 
is conducted without the intervention of an operator. 

20. The method of claim 17, wherein the illuminating step 
further comprises illuminating the region of interest with 50 
light in at least one spectral band which penetrates to the 
papillary dermis and is re-emitted therefrom. 

21. The method of claim 20, wherein the digital imaging 
step further comprises digitally imaging the region of inter- 
est with a digital camera, 55 

22. The method of claim 20, wherein the illuminating step 
further comprises illuminating the region of interest with 
light in a near infrared spectral band. 

23. The method of claim 17, further comprising suppress- 
ing specular reflections prior to the digital imaging step. 50 

24. The method of claim 23, wherein the processor 
converts the digital signals of each of the digital images into 
values corrected for non-uniformities of illumination and of 
response prior to the segmenting step. 

25. The method of claim 17, further comprising: 55 
photographing the region of interest with a color camera 

to form color photographic slides; and 
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illuminating the color photographic slides with light in 
each spectral band; 

wherein the digital imaging step comprises digitally imag- 
ing the illuminated color photographic slides of the 
region of interest with a digital camera, 

26. The method of claim 17, wherein the segmenting step 
further comprises applying the segmentation mask to the 
digital images in the other spectral bands. 

27. The method of claim 17, wherein the segmenting step 
comprises generating the mask at the shortest available 
wavelength. 

28. The method of claim 17, wherein the illuminating step 
comprises illuminating the region of interest by light in at 
least one spectral band whose center is between about 400 
to about 500 nanometers, and the segmenting step comprises 
generating the mask from a digital image at the spectral band 
between about 400 to about 500 nanometers. 

29. The method of claim 19, wherein the segmenting step 
comprises generating the segmentation mask from a digital 
image by: 

removing digital signals from the digital image which 
correspond to hair structures; 

deriving a threshold from a multimodal histogram of 
intensity levels; 

iteratively applying the threshold to the digital signals of 
the digital image; and 

removing digital signals which correspond to small blob- 
like regions from the masked image. 

30. The method of claim 19, wherein the segmenting step 
comprises generating the segmentation mask by comparing 
estimated values which are a function of textures within the 
digital images with a threshold. 

31. The method of claim 30, further comprising generat- 
ing the segmentation mask by comparing the estimated 
texture values to a threshold derived through statistical 
analysis of each digital image. 

32. The method of claim 17, wherein the computing step 
comprises estimating at least one value which is a function 
of the texture of the region of interest determined by the 
segmentation mask. 

33. The method of claim 32, wherein the computing step 
further comprises estimating values which are a function of 
the texture of the region of interest determined by the 
segmentation mask separately in each spectral band, based 
on the same segmentation mask. 

34. The method of claim 32, wherein the computing step 
comprises estimating values which are statistical measures 
of local intensity variation in the digital images in each 
spectral band which are a function of texture. 

35. The method of claim 17, wherein the computing step 
further comprises estimating a value which is a function of 
the asymmetry of the segmented image in each spectral 
band, for two principal axes of the segmented image. 

36. The method of claim 35, wherein the computing step 
further comprises: 

locating the principal axes by computing an orientation 
angle; 

computing the intensity centroid; 

rotating the digital image such that the principal axes are 

parallel to the image axes; 
estimating asymmetry values for each principal axis based 

on the intensity centroid; and 
summing the estimated asymmetry values for the two 

principal axes. 

37. The method of claim 36, wherein the computing step 
further comprises computing the intensity moment with a 
binary intensity distribution. 
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38. The method of claim 17, wherein the computing step 51. The system of claim 49, wherein the processor corn- 
further comprises estimating at least one value which is a pares a weighted combination of estimated values against a 
function of the blotchiness of the segmented digital image, threshold value, 

the estimated blotchiness value being defined through sta- 52. The system of claim 45, wherein the camera records 
tistical properties of the spatial distribution of topographic 5 monochromatic images and the illumination means corn- 
regions of the digital images at each spectral band. prises - 

39. The method of claim 38 wherein the computing step a m M ,. h , SQUrce wi , h feedback tQ stabi , jze 

further comprises determining the centroids of topographic .u • ♦ •* • u i *u u j 

f4l 5 t , .f t . . * i i the intensity in each wavelength band; 

regions of the segmented digital image at each spectral band. J . . & ( 

40. The method of claim 17, wherein the characterizing in means for sequentially filtenng the light; and 

step comprises comparing a weighted combination of esti- an °P llcal flber nn S illuminator to distnbute the filtered 

mated values against a threshold value. hght. 

41. The method of claim 40, wherein the condition of the 53 - The svstem of claim 45, further comprising a feedback 
region of interest to be characterized is the presence of a 1°°P for stabilizing the intensity of the light source by the 
melanoma and weight coefficients for each estimated value processor. 

and the threshold value are selected to maximize specificity, 54 * ^ svslem of claim 45 ' wherein the filter means 

under the constraint of a defined sensitivity to melanoma, on comprises a plurality of interference filters mounted on a 

a representative set of training images. whee l for stepping any filter into a position intercepting the 

42. The method of claim 17, where the characterization li Z hi from the U S hl source * 

step is based on a non-linear combination of the estimated 55 - The system of claim 45, wherein at least one of the 

va j ues zu spectral bands has a center which is between about 400 to 

43. The method of claim 17, where the characterization about 500 nanometers, and at least one other band centered 
step is based on a linear combination of the estimated values. elsewhere in the visible region. 

44. The method of claim 17, where the characterization 56 * The system of claim 55, wherein the set of interfer- 
step is based on a sequential combination of applying a ence filters includes a filter whose center lies in at least one 
linear combination of the estimated values and a non-linear s P ectra l band in the near infrared ran & e wh <>se center lies 
combination of estimated values. between about 750 and 1000 nanometers. 

45. A system for characterizing the condition of a region 57 ■ ^ svstem of ciaim 50 > wherein the camera is a 
of interest of skin, comprising: single-chip, charge-coupled device and the control means 
a source of illumination of light in at least three spectral „ comprises a digital computer including means for determin- 

j )anc j s . JU ing exposure times for the camera which maximize the 

a camera for acquiring digital images of the region of signal-to-noise ratio in the image at each spectral band, 

interest based on the light re-emitted from the illuminated 58 - ^ s y stem of claira 45 > wherein: 

region of interest at each of the spectral bands, the digital the source of illumination provides broad-band ("white") 

image comprising digital signals whose values are a „ 

light; and 

function of the condition of the region of interest; the camera comprises multiple charge-coupled devices 

memory for storing the digital images provided by the which detect light in a plurality of spectral bands 

camera; between the near ultraviolet to near infrared, 

a digital processor programmed to perform the steps of: 59. The system of claim 45, wherein the processor esti- 

segmenting the digital images stored in memory by gen- 40 mates values which are statistical measures of local intensity 

erating a single segmentation mask, where the single variation in the digital images in each spectral band, which 

segmentation mask is the segmentation mask having are a function of the texture of the region of interest, 

largest area of segmentation masks generated from 60 - The system of claim 45, wherein the processor esti- 

each image in each of the at least three spectral bands; mates values based on the ratio of standard deviation of the 

estimating at least one value for each digital image at each 45 areas of dermal papillae to their mean within the segmen- 

spectral band which is a function of the texture of the tation mask. 

portion of the region of interest determined by the 6L llie svstem of claim 45 ' wherein the processor esti- 

segmentation mask* mates values of the average and standard deviation of the 

characterizing the condition of the skin based on the ihi ^ nQ ?J> of rete rid f * with ! n the segmentation masks. 

estimated values; and 50 62 ' ™ e SySlem ° f f lauI \ 45 ; wherem the P«ff° r estl ' 

, t , , t . 4 . f . c . t . mates at least one value which is a function of the asym- 

outputting the characterization of the region of interest. , PtU . , , * u ♦ 1 u ^ f . 

as m . ri* , iU . . c metry or the region or interest in each spectral band, for two 

46. The system of claim 45, further comprising means tor . . , c , , . , 

. 3 . n r .u r * * principal axes of the segmented image by: 

suppressing specular reflections from the region or interest. , . , • . , 1 . 

47. The system of claim 45, further comprising means for locatin S the P nnci P al axes by com P utin g an onentation 
calibrating each digital image to provide correction for 55 an & e ' 

non-uniformities of illumination and response. computing the intensity centroid; 

48. The system of claim 45, wherein the digital processor rotating the digital image such that the principal axes are 
is coupled to the source of illumination and to the camera for parallel to the image axes; and 

controlling the intensity of illumination and exposure times, estimating asymmetry values for each principal axis based 

respectively. 60 on the intensity centroid; and 

49. The system of claim 45, wherein the processor applies summing the estimated asymmetry values for the two 
the segmentation mask derived from the digital images at principal axes. 

one spectral band to the digital images at the other spectral 63. The system of claim 45, wherein the processor further 

bands. estimates at least one value which is a function of the 

50. The system of the claim 45, where in the processor 65 blotchiness of the region of interest. 

estimates values separately from digital images at each 64. The system of claim 45, wherein the processor further 

spectral band based on the segmentation mask. estimates at least one value which is a function of the 
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irregularity of the border of the region of interest by esti- 
mating a value which is a statistical measure of the deviation 
of the border of the segmentation mask from the border of 
an ellipse of the same area, aspect ratio, and orientation as 
the segmentation mask. 5 

65. The system of claim 45, wherein the processor further 
estimates a value which is a function of the gradient at the 
border of the region of interest by estimating a statistical 
measure of the gradient values of the intensity of the digital 
images across the border of the segmented images, at each 3Q 
spectral band. 

66. The system of claim 45, wherein the processor char- 
acterizes the type of lesion as invasive or non-invasive. 

67. The system of claim 45, wherein the processor gen- 
erates the segmentation mask from a digital image by: 

removing digital signals from the digital image which 15 
correspond to hair structures; 

deriving a threshold from a multimodal histogram of 
intensity levels; 

iteratively applying the threshold to the digital signals of ^ 
the digital image; and 

removing digital signals which correspond to small blob- 
like regions from the masked image, 

68. A system for characterizing the condition of a region 

of interest of skin, comprising: 25 
a source of illumination of light in at least three spectral 
bands; 

a camera for acquiring digital images of the region of 
interest based on the light re-emitted from the illuminated 
region of interest at each of the at least three spectral 3Q 
bands, the digital image comprising digital signals whose 
values are a function of the condition of the region of 
interest; 

a memory for storing the digital images; 

a digital processor including: 35 
digital processing means for segmenting the digital 
images stored in memory and computing estimated 
values of parameters which are a function of the 
segmented images, wherein the digital images are seg- 
mented by generating a single segmentation mask, 40 
where the single segmentation mask is the segmenta- 
tion mask having largest area of segmentation masks 
generated from each image in each of the at least three 
spectral bands; 

digital processing means for automatically characterizing 45 
the condition of the tissue based on the estimated 
values; and 

means for outputting the characterization of the region of 
interest. 

69. A method of characterizing the condition of a region 50 
of interest of skin, wherein the absorption and scattering of 
light in different spectral bands by the region of interest is a 
function of the condition of the skin, the method comprising: 
illuminating a portion of the skin including the region of 

interest by light in at least three spectral bands; 55 
digitally imaging the portion of the skin including the region 
of interest at the at least three spectral bands with the light 
re-emitted by the portion of the skin to generate digital 
images comprising digital signals whose values are a 
function of the condition of the region of interest of the 60 
skin; and 

providing the digital images to a processor, wherein the 
processor: 

segments the digital images by generating a segmentation 
mask defining the boundary of the region of interest 65 
from a digital image in any one of the at least three 
spectral bands; 
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computes at least one estimated value which is a statistical 
measure of the deviation of the boundary of the region 
of interest from the boundary of an ellipse of the same 
area, aspect ratio, and orientation as the segmentation 
mask; 

characterizes the condition of the region of interest of the 
skin based on the estimated values; and 

outputs the characterization of the condition of the region 
of interest of the skin. 

70. A method of characterizing the condition of a region 
of interest of skin, wherein the absorption and scattering of 
light in different spectral bands by the region of interest is a 
function of the condition of the skin, the method comprising: 
illuminating a portion of the skin including the region of 

interest by light in at least three spectral bands; 
digitally imaging the portion of the skin including the region 
of interest at the at least three spectral bands with the light 
re-emitted by the portion of the skin to generate digital 
images comprising digital signals whose values are a 
function of the condition of the region of interest of the 
skin; and 

providing the digital images to a processor, wherein the 
processor: 

segments the digital images by generating a segmentation 
mask defining the boundary of the region of interest 
from a digital image in any one of the at least three 
spectral bands; 

computes at least one estimated value of a statistical 
measure of the gradient values of the intensity of the 
digital images across the border of the segmented 
images; 

characterizes the condition of the region of interest of the 
skin based on the estimated values; and 

outputs the characterization of the condition of the region 
of interest of the skin. 

71. A method of characterizing the condition of a region 
of interest of skin, wherein the absorption and scattering of 
light in different spectral bands by the region of interest is a 
function of the condition of the skin, the method comprising: 
illuminating a portion of the skin including the region of 

interest by light in at least three spectral bands; 
digitally imaging the portion of the skin including the region 
of interest at the at least three spectral bands with the light 
re-emitted by the portion of the skin to generate digital 
images comprising digital signals whose values are a 
function of the condition of the region of interest of the 
skin; and 

providing the digital images to a processor, wherein the 
processor: 

segments the digital images by generating a segmentation 
mask defining the boundary of the region of interest 
from a digital image in any one of the at least three 
spectral bands; 

computes at least one estimated value based on the ratio 
of standard deviation of the areas of dermal papillae to 
their mean within the segmentation mask; 

characterizes the condition of the region of interest of the 
skin based on the estimated values; and 

outputs the characterization of the condition of the region 
of interest of the skin. 

72. A method of characterizing the condition of a region 
of interest of skin, wherein the absorption and scattering of 
light in different spectral bands by the region of interest is a 
function of the condition of the skin, the method comprising: 
illuminating a portion of the skin including the region of 

interest by light in at least three spectral bands; 
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digitally imaging the portion of the skin including the region 
of interest at the at least three spectral bands with the light 
re -em it ted by the portion of the skin to generate digital 
images comprising digital signals whose values are a 
function of the condition of the region of interest of the 5 
skin; and 

providing the digital images to a processor, wherein the 
processor: 

segments the digital images by generating a segmentation 
mask defining the boundary of the region of interest 10 
from a digital image in any one of the at least three 
spectral bands; 
computes at least one estimated value of the average and 
standard deviation of the thickness of rete ridges within 
the segmentation mask for a digital image of the region 15 
of interest determined by the segmentation mask; 
characterizes the condition of the region of interest of the 

skin based on the estimated values; and 
outputs the characterization of the condition of the region 2 o 
of interest of the skin for a digital image of the region 
of interest determined by the segmentation mask. 
73. A method of characterizing the condition of a region 
of interest of skin, wherein the absorption and scattering of 
light in different spectral bands by the region of interest is a 2 s 
function of the condition of the skin, the method comprising: 
illuminating a portion of the skin including the region of 
interest by light in at least three spectral bands; 
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digitally imaging the portion of the skin including the region 
of interest at the at least three spectral bands with the light 
re-emitted by the portion of the skin to generate digital 
images comprising digital signals whose values are a 
function of the condition of the region of interest of the 
skin; 

calibrating each pixel location in the digital image in each 
spectral band with respect to stored images of a white 
target material having known diffuse reflectance, each of 
the stored images being an average of a plurality of 
images acquired at each spectral band, while the material 
undergoes continual in-plane motion; and 

providing the digital images to a processor, wherein the 
processor: 

segments the digital images by generating a segmentation 
mask defining the boundary of the region of interest 
from a digital image in any one of the at least three 
spectral bands; 

computes at least one estimated value for each digital 
image at each spectral band which is a function of a 
characteristic of the region of interest determined by 
the segmentation mask; 

characterizes the condition of the region of interest of the 
skin based on the estimated values; and 

outputs the characterization of the condition of the region 
of interest of the skin. 

* * * * * 
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description of any pediatric subpopulations that suffer from the disease or condition that the device is intended to 
treat, diagnose, or cure; and (2)the number of affected pediatric patients. 

In future premarket submissions, we encourage you to provide (or continue to provide) an electronic copy of 
your submission. By doing so, you will save FDA resources and may help reviewers navigate through Jonger 
documents more easily. Under CDRH's eCopies Program, you may replace one paper copy of any premarket 
submission (e.g., 5 1 0(k), IDE, PMA, HDE) with an electronic copy. For more information about the program, 
including the formatting requirements, please see www.fda.gov/cdrh/elecsub.html. 



Sincerely yours, 





Pauline Foeartv 

Consumer Safety Officer 

Division of Surgical, Orthopedic and Restorative Devices 

Office of Device Evaluation 

Center for Devices and Radiological Health 




DEPARTMENT OF HEALTH & HUMAN SERVICES 



Public Health Service 



Electro-Optical Sciences, Inc. (EOS) 
% Joseph V. Gulfo, M.D., M.B.A 
President and CEO 
1 Bridge Street, Suite #11 
Irvington, New York 1 0533 

Re: P090012 
Melafind 

Dear Dr. Gulfo: 

The Center for Devices and Radiological Health (CDRH) of the Food and Drug Administration 
(FDA) has completed an initial review of your premarket approval application (PMA). We 
inadvertently made an error in your Filing Letter which stated an incorrect July 9, 2009 Filing 
Date. The correct Filing date is June 9, 2009. 

We hope that this incorrect Filing Date has not inconvenienced you. If you have any questions 
about this corrective action, please contact Atiq Chowdhury at (301) 796-6391 . 

Sincerely yours, 

Mark N. Melkerson 
Director 

Division of Surgical, Orthopedic, 
and Restorative Devices 
Office of Device Evaluation 
Center for Devices and 
Radiological Health 



Food and Drug Administration 
9200 Corporate Boulevard 
Rockvilfe MD 20850 
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Food and Drug Administration 
9200 Corporate Boulevard 
Rockville MD 20850 

Electro-Optical Sciences, Inc. (EOS) |m 0 Q 

% Joseph V. Gulfo, M.D., M.B.A JUL ^ 9 2009 

President and CEO 

1 Bridge Street, Suite #11 

Irvington, New York 10533 

Re: P090012 
Mclafind 

Filed: July 9, 2009 
Dear Dr. Gulfo: 

The Center for Devices and Radiological Health (CDRH) of the Food and Drug Administration 
(FDA) has completed an initial review of your premarket approval application (PMA). We are 
pleased to inform you that we have made a threshold determination that the PMA is sufficiently 
complete to permit a substantive review and is, therefore, suitable for filing. The filing date is 
July 9, 2009, which is the date of CDRH receipt of the PMA. 

We are also pleased to inform you that your application will receive expedited processing. 
Expedited review status was granted for the following reasons: 

1 . The MelaFind device described in your application does meet the criteria of a device 
intended to affect a condition that is life-threatening and is irreversibly debilitating. 
Early or better detection of melanoma has the potential to reduce both life altering 
surgeries and reduce mortality due to melanoma. 

2. Use of the device has potential significant benefit to patients by: reducing unnecessary 
biopsies for suspicious melanomas, reducing potential biopsy adverse events such as 
scarring, infection, pain, and potentially improve the detection of early melanomas 
which will improve the survivability of patients diagnosed with melanoma. 

3. The MelaFind device is a breakthrough technology in that it uses multiple 
wavelengths of light to produce multi-wavelength reflected optical signals for lesion 
diagnosis. 

You are reminded that it is imperative that the information used to support an application for 
expedited review meet the requirements of valid scientific evidence (21 CFR 860.7). This 
evidence would generally be obtained from well-designed, -monitored, and -controlled clinical 
trials so that the true merit of the medical device might be evaluated as promptly and efficiently 
as possible. You are further advised that the granting of expedited review status does not 
guarantee that the application will ultimately be approved. 
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Please be advised that the decision to file the PMA does not imply that either an in-depth 
evaluation of the safety and effectiveness of the device has been performed or a decision about 
the approvability of the application has been made. Rather, it represents a decision by CDRH 
that the application is sufficiently complete to begin the substantive review process. Further 
review of your application may result in deficiencies which will be communicated to you. 

Following receipt of a filing letter, an applicant is required by 21 CFR 814.20(e) to update its 
pending PMA 3 months after the filing date with new safety and effectiveness information 
learned about the device from ongoing or completed studies when the information may either 1) 
reasonably affect an evaluation of the safety or effectiveness of the device or 2) reasonably affect 
the statement of contraindications, warnings, precautions and adverse reactions in the draft 
labeling. 

This updated reporting is limited to studies sponsored by the applicant or to which the applicant 
has reasonable access. The update report should be consistent with the data reporting provisions 
of the protocol. Please submit clinical updates in three copies as an amendment to the PMA and 
include the above PMA reference number assigned to the PMA. 

The PMA cannot be approved until FDA has determined that the manufacturing facilities, 
methods and controls comply with the conditions set forth in your application and the applicable 
requirements of the Quality System Regulation (21 CFR Part 820). If you have not already done 
so, please notify CDRH as soon as possible in the form of an amendment to the PMA if there 
will be a delay in setting up your manufacturing facility for production of the device, and provide 
the expected date that the facility will be prepared for an FDA inspection. If you have any 
questions regarding the status of your Quality System inspection please contact the Office of 
Compliance at (301) 796-5815, or your District Office, 

A meeting of The General and Plastic Surgery Devices Panel will be held at which your PMA 
will be reviewed. You will be notified of the location and date of this meeting. Any additional 
information to be included in your PMA should be submitted in the form of a PMA amendment 
and be received by FDA at least 8 weeks in advance of the scheduled advisory panel meeting in 
order for FDA and the panel members to have adequate time to review the new information. 
Information received bv CDRH less than 8 weeks in advance of a scheduled advisory panel 
meeting will not be considered or reviewed at the meeting and may delay consideration of your 
PMA supplement until a subsequent advisory panel meeting . 

For your information, there is an industry representative on this FDA advisory pane! whose 
name, address and telephone number you can obtain by contacting the Committee Management 
Staff at (240) 276-0450. CDRH believes that industry representatives will be better prepared to 
participate in panel discussions if they have been provided with at least a copy of the Summary of 
Safety and Effectiveness Data for review prior to the panel meeting. In accordance with 21 CFR 
14.86(b), all panel members are subject to all rules and regulations adopted by FDA and the 
committee; therefore, even though the industry representatives usually are not given access to 
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trade secret and confidential, commercial information, they are bound to protect the 
confidentiality of documents that would be sent to them in preparation for panel review of a 
PMA. If you would like the industry representative to have access to any portion of your PMA, 
including the Summary of Safety and Effectiveness Data, please provide a copy to FDA for that 
purpose. Clearly identify the submission as a purged copy intended for review by the industry 
representative. Review of your PMA will not be prejudiced if you elect not to provide 
information for industry representative review. 

All correspondence regarding this PMA should be submitted in six (6) copies in the form of a 
PMA amendment. Please address all submissions to: 



PMA Document Mail Center (HFZ-401) 
Center for Devices and Radiological Health 
Food and Drug Administration 
9200 Corporate Blvd. 
Rockville, Maryland 20850 



If you have any questions regarding this letter, please contact Atiq Chowdhury at (301) 796- 



6391. 




Mark K Melkerson 
Director 

Division of Surgical, Orthopedic 
and Restorative Devices 

Office of Device Evaluation 

Center for Devices and 
Radiological Health 



EXHIBIT H 

Letter Substantiating PMA Approval Date 
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Public Health Service 



DEPARTMENT OF HEALTH & HUMAN SERVICES 



Food and Drug Administration 
10903 New Hampshire Avenue 



Document Control Room -WO66-G609 
Silver Spring, MD 20993-0002 
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MELA Sciences, Inc. 

% Joseph V. Gulfo, M.D., MBA 

President and CEO 

50 South Buckhout Street, 

Suite 1, 

Irving NY, 10533 



Re: P090012 
MelaFind® 
Filed: June 9, 2009 

Amended: September 24, October 27, 2009; March 29 (2), April 1 6 and 23, May 1 0 and 26, 
August 5, September 8 and 13, 2010; February 22, 2011; May 12, 201 1; August 23 and 31; 
201 1; September 22, 201 1; October 6, 201 1 and November 1, 201 1 . 
Procode: OYD 

DearDr.Gulfo: 

The Center for Devices and Radiological Health (CDRH) of the Food and Drug Administration 
(FDA) has completed its review of your premarket approval application (PMA) for the MelaFind 
device, 

MelaFind is intended for use on clinically atypical cutaneous pigmented lesions with one or 
more clinical or historical characteristics of melanoma, excluding those with a clinical 
diagnosis of melanoma or likely melanoma, MelaFind is designed to be used when a 
dermatologist chooses to obtain additional information for a decision to biopsy. MelaFind 
should NOT be used to confirm a clinical diagnosis of melanoma. 

MelaFind is only for use by physicians trained in the clinical diagnosis and management of skin 
cancer (i.e., dermatologists) who have also successfully completed a training program in the 
appropriate use of MelaFind. 

The MelaFind result is one element of the overall clinical assessment. MelaFind positive 
lesions (which may include malignant melanoma, melanoma in situ, high grade dysplastic nevi 
and atypical melanocyte proliferation/hyperplasia) should be considered for biopsy; the biopsy 
decision of a MelaFind negative lesion should be based on the remainder of the entire clinical 
context. Lesions that are "non-evaluable" by MelaFind should be carefully re-evaluated for 
biopsy. 

MelaFind is indicated only for use on lesions with a diameter between 2 mm and 22 mm, 
lesions that are accessible by the MelaFind imager, lesions that are sufficiently pigmented (i.e. 
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not for use on non-pigmented or skin-colored lesions), lesions that do not contain a scar or 
fibrosis consistent with previous trauma, lesions where the skin is intact (i.e., non-ulcerated or 
non-bleeding lesions), lesions greater than 1 cm away from the eye, lesions which do not 
contain foreign matter, and lesions not on special anatomic sites (i.e., not for use on acral, 
palmar, plantar, mucosal, or subungual areas). MelaFind is not designed to detect pigmented 
non-melanoma skin cancers, so the dermatologist should rely on clinical experience to 
diagnose such lesions. 

We are pleased to inform you that the PMA is approved. You may begin commercial distribution 
of the device in accordance with the conditions of approval described below. You may continue 
commercial distribution of the device upon receipt of this letter. 

The sale and distribution of this device are restricted to prescription use in accordance with 21 
CFR 801.109 and under section 51 5(d)(l)(B)(ii) of the Federal Food, Drug, and Cosmetic Act (the 
act). The device is further restricted under section 515(d)(l)(B)(ii) of the act insofar as the labeling 
must specify the specific training or experience practitioners need in order to use the device. FDA 
has determined that these restrictions on sale and distribution are necessary to provide reasonable 
assurance of the safety and effectiveness of the device. Your device is therefore a restricted device 
subject to the requirements in sections 502(q) and (r) of the act, in addition to the many other FDA 
requirements governing the manufacture, distribution, and marketing of devices. 

Continued approval of this PMA is contingent upon the submission of periodic reports, required 
under 21 CFR 814.84, at intervals of one year (unless otherwise specified) from the date of 
approval of the original PMA. Two copies of this report, identified as " Annual Report " (please use 
this title even if the specified interval is more frequent than one year) and bearing the applicable 
PMA reference number, should be submitted to the address below. The Annual Report should 
indicate the beginning and ending date of the period covered by the report and should include the 
information required by 21 CFR 814.84. 

In addition to the above, and in order to provide continued reasonable assurance of the safety and 
effectiveness of the device, the Annual Report must include, separately for each model number (if 
applicable), the number of devices sold and distributed during the reporting period, including those 
distributed to distributors. The distribution data will serve as a denominator and provide necessary 
context for FDA to ascertain the frequency and prevalence of adverse events, as FDA evaluates the 
continued safety and effectiveness of the device. 

In addition to the Annual Report requirements, you must provide the following data in post- 
approval study reports (PAS). Two copies, identified as " PMA Post-Approval Studv Report " and 
bearing the applicable PMA reference number, should be submitted to the address below. 

1 . You must conduct a post approval study that will evaluate whether MelaFind increases the 
sensitivity of physicians in diagnosing melanomas and high-grade lesions, while the false 
positive rate of physicians is not substantially elevated. 
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2. The study will be a multi-center, single arm, observational, prospective study to gather data 
on relative sensitivity, among other study endpoints. Data to be collected includes: relative 
sensitivity comparing physicians' performance before and after using MelaFind as the 
primary study endpoint; real-world use of MelaFind, i.e., the patient characteristics 
including age, gender, race/ethnicity, and Fitzpatrick Skin Type, the number of lesions that 
were examined by MelaFind, the proportion of lesions that meet the labeled Indications For 
Use among all the lesions examined by MelaFind, the proportions of positive and negative 
findings of MelaFind among all of the lesions examined, the proportion of lesions that are 
un-evaluable by MelaFind, the proportion of lesions that are found to be un-evaluable for 
each user of MelaFind, the number of attempts widi MelaFind that were performed for each 
lesion before a definitive reading resulted or the lesion was declared un-evaluable, and the 
impact of MelaFind use on the per physician biopsy rate for pigmented lesions; and an 
evaluation of safety and effectiveness of MelaFind, i.e., the proportion of biopsy from the 
lesions that MelaFind identifies as positive and the results of those biopsies, the proportion 
of biopsy among the "unreadable" lesions and the results of those biopsies, the proportion 
of biopsy from the lesions that MelaFind identifies as negative and the results of those 
biopsies, and the proportion of the biopsied lesions (from each of the above - MelaFind 
positive, MelaFind negative, and un-evaluable) returned as melanoma on pathology. This 
study must enroll 78 patients with one or more eligible and evaluable histologically- 
confirmed melanoma and/or high-grade lesion based on the null hypothesis that the relative 
sensitivity is less than or equal to 1.1. The study power will be at least 85%. 

3. Patients with lesions evaluated with MelaFind during the enrollment period, but not 
biopsied at that time, will be followed at 1 year ±3 months and 2 years ±3 months. At least 
50% of the study sites will be new (i.e., they did not participate in the MelaFind pivotal 
study). The study will enroll a mix of academic centers and private practices. 

Please be advised that the results from these studies should be included in the labeling as these data 
become available. Any updated labeling must be submitted to FDA in the form of a PMA 
Supplement. 

FDA would like to remind you that you are required to submit separate PAS Progress Reports 
every six months during the first two years and annually thereafter. The reports should clearly be 
identified as Post-Approval Study Report. Two copies for each study, identified as " PMA Post- 
A pproval Study Report " and bearing the applicable PMA reference number, should be submitted to 
the address below. For more information on post-approval studies, see the FDA guidance 
document entitled, "Procedures for Handling Post- Approval Studies Imposed by PMA Order" 
(wA\wTda,gov/MedicalDevices/Device^ 
tm#2). 

Be advised that the failure to conduct any such study in compliance with the good clinical 
laboratory' practices in 21 CFR part 58 (if a non-clinical study subject to part 58) or the institutional 
review board regulations in 21 CFR part 56 and the informed consent regulations in 21 CFR part 
50 (if a clinical study involving human subjects) may be grounds for FDA withdrawal of approval 
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ofthePMA. 

Within 30 days of your receipt of this letter, you must submit a PMA supplement that includes a 
complete protocol of your post-approval study. Your PMA supplement should be clearly labeled 
as a "Post-Approval Study Protocol" and submitted in triplicate to the address below. Please 
reference the PMA number above to facilitate processing. If there are multiple protocols being 
finalized after PMA approval, please submit each protocol as a separate PMA supplement. For 
more information on post-approval studies, see the FDA guidance document entitled, "Procedures 
for Handling Post- Approval Studies Imposed by PMA Order" 
(www,fda,gov/MedicalDev 
gn#2). 

Before making any change affecting the safety or effectiveness of the device, you must submit a 
PMA supplement or an alternate submission (30-day notice) in accordance with 21 CFR 8 1439. 
All PMA supplements and alternate submissions (30-day notice) must comply with the applicable 
requirements in 21 CFR 814.39. For more information, please refer to the FDA guidance 
document entitled, "Modifications to Devices Subject to Premarket Approval (PMA) - The PMA 
Supplement Decision-Making Process" 
( w\\w.fda,gov/MedicalDevice^ 

tm)- 

You are reminded that many FDA requirements govern the manufacture, distribution, and 
marketing of devices. For example, in accordance with the Medical Device Reporting (MDR) 
regulation, 21 CFR 803.50 and 21 CFR 803.52, you are required to report adverse events for this 
device. Manufacturers of medical devices, including in vitro diagnostic devices, are required to 
report to FDA no later than 30 calendar days after the day they receive or otherwise becomes aware 
of information, from any source, that reasonably suggests that one of their marketed devices: 

1. May have caused or contributed to a death or serious injury; or 

2. Has malfunctioned and such device or similar device marketed by the manufacturer 
would be likely to cause or contribute to a death or serious injury if the malfunction 
were to recur. 

Additional information on MDR, including how, when, and where to report, is available at 
www.fda.gov/MedicalDe\nces/SafeW/ReportaProblem/default.htm . 

In accordance with the recall requirements specified in 21 CFR 806.10, you are required to submit 
a written report to FDA of any correction or removal of this device initiated by you to: (1) reduce a 
risk to health posed by the device; or (2) remedy a violation of the act caused by the device which 
may present a risk to health, with certain exceptions specified in 21 CFR 806.10(a)(2). Additional 
information on recalls is available at w f ww < fda^ov/Safetv/Recalls/lndustrvGuidance/default.htm , 

CDRH does not evaluate information related to contract liability warranties. We remind you; 
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however, that device labeling must be truthful and not misleading. CDRH will notify the public of 
its decision to approve your PMA by making available, among other information, a summary of the 
safety and effectiveness data upon which the approval is based. The information can be found on 
the FDA CDRH Internet HomePage located at 
uw.fda,gov/MedicalDevices#^ 

MAApprovals/defaulthtm . Written requests for this information can also be made to the Food and 
Drug Administration, Dockets Management Branch, (HFA-305), 5630 Fishers Lane, Rm. 1 061, 
Rockville, MD 20852. The written request should include the PMA number or docket number. 
Within 30 days from the date that this information is placed on the Internet, any interested person 
may seek review of this decision by submitting a petition for review under section 5 1 5(g) of the act 
and requesting either a hearing or review by an independent advisory committee. FDA may, for 
good cause, extend this 30-day filing period. 

Failure to comply with any post-approval requirement constitutes a ground for withdrawal of 
approval of a PMA. The introduction or deliveiy for introduction into interstate commerce of a 
device that is not in compliance with its conditions of approval is a violation of law. 

You are reminded that, as soon as possible and before commercial distribution of your device, you 
must submit an amendment to this PMA submission with copies of all approved labeling in final 
printed form. Final printed labeling that is identical to the labeling approved in draft form will not 
routinely be reviewed by FDA staff when accompanied by a cover letter stating that the final 
printed labeling is identical to the labeling approved in draft form. If the final printed labeling is 
not identical, any changes from the final draft labeling should be highlighted and explained in the 
amendment. 

All required documents should be submitted in triplicate, unless otherwise specified, to the address 
below and should reference the above PMA number to facilitate processing. One of those three 
copies may be an electronic copy (eCopy), in an electronic format that FDA can process, review 
and archive (general information: 
http;//w^vw.fda.gov/Medic^ 

marketSubmissions/ucml34508.htm : clinical and statistical data: 

http://www i ,fda.gov/Medi^ 

marketSubmissions/ucm 1363 77.html 

U.S. Food and Drug Administration 
Center for Devices and Radiological Health 
PMA Document Mail Center - WO66-G609 
10903 New Hampshire Avenue 
Silver Spring, MD 20993-0002 
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If you have any questions concerning this letter, please contact Aliq Chowdhuty at 301-796-6391. 

Sincerely yours, 

Christy Foreman 
Director 

Office of Device Evaluation 

Center for Devices and Radiological Health 

Food and Drug Administration 
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MelaFind System Regulatory Summary 



Description 


Date 


Originator 


Letter to IDE Staff seeking confirmation that the MelaFind 
system does not require an approved Investigational Device 
Exemption (IDE) as a significant risk device and may obtain an 
approved IDE as a not significant risk device 


8/17/2000 


MELA 1 


Email requesting additional information in response to MELA's 
8/17/00 inquiry 


8/30/00 


FDA 


Email responding to FDA's 8/30/00 request for additional 
information 


9/19/00 


MELA 


Email requesting additional information 


9/20/00 


FDA 


Meeting with FDA, HCFA, and MELA to discuss various 
regulatory aspects of the MelaFind system, including FDA's 
confirmation that a PMA and advisory panel are required 


2/13/01 


MELA 


IRB Approval for Protocol 2001 1 - Patient Examination with 
MelaFind System Developed by EOS 


5/2/01 


Western IRB 
(WIRB) 


IRB Approval for Protocol 20012 - Non-invasive Breslow 
Thickness Measurement for Cutaneous Melanoma with 
MelaMeter 


11/16/01 


WIRB 


Submission to address FDA's concern about the phototoxicity of 
the device for the agency's review of the not significant risk 
nature of the MelaFind system 


3/18/02 


MELA 


Letter stating information from 8/17/00 and 3/18/02 submissions 
does not clearly indicate how the information from the MelaFind 
system will be used in the clinical investigation and therefore 
FDA cannot make a determination of whether the system poses a 
significant risk 


4/1/02 


FDA 


Email responding to FDA's 4/1/02 letter explaining the two- 
phase clinical study plan: (1) a data collection phase and (2) the 
clinical trial phase 


4/8/02 


MELA 


Email stating that the study is non-significant risk device trial 


4/9/02 


FDA 



1 MELA Sciences, Inc. was previously Electro-Optical Sciences, Inc. However, for purposes of this Regulatory 
Summary, we use "MELA" to refer to both. 



LIBW/1 808372.1 



Description 


Date 


Originator 


PMA Shell for Modular Submission 


1 1/7/02 


MELA 


PMA Modular Submission on manufacturing information 


3/4/03 


MELA 


M020024/M2: Acceptance of PMA Modular Submission on 
manufacturing information; module is considered closed 


7/17/03 


FDA 


Request for a meeting to discuss the clinical module (M3) of the 
PMA 


8/15/03 


MELA 


Conference call to discuss possible claims and labeling for the 
MelaFind system 


11/18/03 


MELA 


Submission of Protocol 20031 to FDA 


1 1/24/03 


MELA 


Comments on Protocol 2003 1 


6/1/04 


FDA 


Meeting to discuss Protocol 2003 1 


6/4/04 


MELA 


Submission of Protocol 20031, revised to address FDA's 
comments during 6/4/04 meeting 


6/21/04 


MELA 


Email with comments on Protocol 2003 1 


8/5/04 


FDA 


Meeting to discuss Protocol 2003 1 


8/6/04 


MELA 


Submission of Protocol 20031, revised to address FDA's 
comments in 8/5/04 email and 8/6/04 meeting 


9/2/04 


MELA 


IRB Approval for Protocol 20031 - Evaluation of Pigmented 
Skin Lesions with MelaFind System [20031 -A] 


9/24/04 


WIRB 


Meeting to discuss Protocol 20031 and signing of the Binding 
Protocol Agreement under FD&C Act § 520(g) 


10/20/04 


MELA 


Letter explaining the nature of the difficulties that were 
experienced with the MelaFind system during clinical trial 
2003 1 and plans to address the issues prior to the start of pivotal 
trial 20061 


6/13/05 


MELA 
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Description 


Date 


Originator 


M020024/M1: Acceptance of PMA Modular Submission on 
device design, pre-clinical data used to support the device 
design, software design, and certification to various international 
and national standards; module is considered closed. States 
when software algorithm is completed, company should submit 
additional information to this module or in the PMA. 


7/16/05 


FDA 


Email stating the revised protocol 20061 "does not alter the 
pivotal study design" and the "changes do not alter the signed 
Study Agreement," i.e., the Binding Protocol Agreement applied 
to Protocol 20061 


1/4/06 


FDA 


IRB Approval for Protocol 2003 1-B - Pilot Roll-in Study for 
Protocol 20061 : Evaluation of Pigmented Skin Lesions with 
MelaFind System 


5/3/06 


WIRB 


Request for expedited review 


8/10/06 


MELA 


Expedited Review granted 


10/4/06 


FDA 


IRB Approval for Protocol 20061 - Evaluation of Pigmented 
Skin Lesions with MelaFind System 


4/24/07 


WIRB 


First accrual for Protocol RCP2007-05 - Benign Pigmented Skin 
Lesions: Melanin Localization and Quantification with MelaFind 
System - study sponsored by L'Oreal and conducted in Austria 
and Switzerland and relied on in the PMA 


9/26/07 


L'Oreal 


Submission of PMA for the MelaFind system 


6/9/09 


MELA 


Acknowledgement of PMA P090012 


6/12/09 


FDA 


Filing Letter for PMA P090012 


7/29/09 


FDA 


Corrected Filing Letter for PMA P090012 - stating the correct 
Filing Date is June 9, 2009 


8/18/09 


FDA 


Pre-Approval Inspection; 483 issued 


8/10-13, 

17-18, 

21/09 


FDA 


Response to 483 


9/2/09 


MELA 


FDA emails list of concerns related to the PMA 


9/15/09 


FDA 
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Description 


Date 


Originator 


"100 Day" PMA Review Meeting 


9/17/09 


FDA 


PMA Amendment P090012/A001 - submission of 9/17/09 
meeting minutes and request that FDA immediately set a date for 
the panel meeting 


9/24/09 


MELA 


Deficiency Letter (emailed to company on 10/13/09) 


10/7/09 


FDA 


2 nd response to 483 


10/9/09 


MELA 


PMA Amendment P090012/A002- responding to Deficiency 
Letter 


10/27/09 


MELA 


Establishment Inspection Report (EIR) issued to MELA for 
August 2009 inspection 


1 1/2/09 


FDA 


FDA emails the company and states "it looks like we are heading 
toward a March date" for the panel meeting 


11/23/09 


FDA 


Letter responding to MELA's 483 responses 


12/11/09 


FDA 


Letter responding to FDA's 12/1 1/09 letter and providing an 
update to the 483 responses 


1/8/10 


MELA 


Letter providing an update to MELA's 1/8/10 letter (483 
responses) 


3/1/10 


MELA 


Not Approvable Letter 


3/10/10 


FDA 


PMA Amendment P090012/A003 and A004 [same PMA 
Amendment logged in twice by FDA] - requesting extension 
until 7/2/10 to submit a petition under FD&C Act § 515(d)(4) for 
review under § 5 15(g) of the Not Approvable Letter if company 
elects to treat not approvable letter as a PMA denial 


3/29/10 


MELA 


Request for Supervisory Review under 21 CFR § 10.75 by 
CDRH Director Dr. Jeffrey Shuren of Not Approvable Letter 


4/15/10 


Goodwin 
Procter (on 
behalf of 
MELA) 


PMA Amendment P090012/A005 - response to Not Approvable 
Letter 


4/16/10 


MELA 


Letter providing an update to MELA's 1/8/10 and 3/1/10 letters 
(483 responses) 


4/19/10 


MELA 
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Description 


Date 


Originator 


PMA Amendment P090012/A006 - withdrawal of 
P090012/A005 in order to facilitate the 4/27/10 meeting 


4/23/10 


MELA 


Request that FDA hold in abeyance 4/15/10 request for 
supervisory review in order to facilitate meeting with review 
staff on 4/27/10 


4/23/10 


Goodwin 
Procter (on 
behalf of 
MELA) 


Meeting to discuss Not Approvable Letter 


4/27/10 


MELA 


PMA Amendment P090012/A007 - Response to Not 
Approvable Letter 


5/10/10 


MELA 


PMA Amendment P090012/A008 - notifying FDA of name 
change from Electro-Optical Sciences, Inc. to MELA Sciences, 
Inc. 


5/26/10 


MELA 


Federal Register notice announcing 8/26/10 panel meeting 


6/24/10 


FDA 


Submission of Sponsor's draft panel package 


7/13/10 


MELA 


Postponement of planned 8/26/10 panel meeting 


7/16/10 


FDA 


PMA Amendment P090012/A009 - submission of data and 
analyses for Protocol 20063 Reader Study 


8/5/10 


MELA 


Federal Register notice announcing 11/18/10 panel meeting 


8/16/10 


FDA 


Meeting to facilitate preparations for November panel meeting 


8/26/10 


MELA 


Teleconference to review final indications for use 


9/3/10 


MELA 


PMA Amendment P090012/A010 - submission of revised claim 


9/8/10 


MELA 


PMA Amendment P090012/A01 1 - submission of 8/26/10 
meeting minutes and explanation of revised claim 


9/13/10 


MELA 


Submission of Sponsor's revised panel package 


9/27/10 


MELA 


Federal Register notice correcting 8/16/10 notice 


10/5/10 


FDA 


Submission of Sponsor's final panel package 


10/6/10 


MELA 


Agency's draft Executive Summary 


10/18/10 


FDA 
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Description 


Date 


Originator 


Submission of comments on agency's draft Executive Summary 
and addendum to Sponsor's Executive Summary 


10/20/10 


MELA 


Agency's revised Executive Summary 


10/21/10 


FDA 


Submission of comments on agency's revised Executive 
Summary and revised addendum to Sponsor's Executive 
Summary 


10/22/10 


MELA 


General and Plastic Surgery Devices Panel Meeting considering 
the PMA 


11/18/10 


FDA 


Teleconference with FDA in which MELA requests a post Panel 
meeting 


12/3/10 


MELA 


PMA Amendment - P090012/A012 - modifying the indications 
for use statement to define the intended users of the device as 
dermatologists 


2/22/1 1 


MELA 


Submission of Citizen Petition asking that Commissioner 
Hamburg enforce the Oct. 20, 2004 Binding Protocol Agreement 
and require that MELA's PMA is reviewed fairly and 
consistently with FDA's laws and regulations 


5/9/1 1 


MELA 


PMA Amendment P090012/A013 - for the draft MelaFind 
System User Training Program 


5/12/11 


MELA 


During House Subcommittee for Oversight and Investigation 
hearing, CDRH Director Dr. Jeffrey Shuren states FDA staff 
incorrectly failed to send PMA to advisory panel 


7/20/1 1 


FDA 


Letter identifying issues that warrant additional discussion 
between the company and FDA to reach an approvable decision 


8/11/11 


FDA 


PMA Amendment P090012/A014 - Identifying areas of 
substantial disagreement with FDA's 8/1 1/1 1 letter 


8/23/11 


MELA 


PMA Amendment P090012/A015 - Full response to FDA's 
8/11/11 letter 


8/31/11 


MELA 


Meeting to get PMA in approvable form 


9/7/1 1 


FDA 
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Description 


Date 


Originator 


PMA Amendment P090012/A016 - Patient Labeling, User 
Guide, Package Insert, Training Program, Winter Clinical 
Reader Study, Slides from 9/7/1 1 meeting, Post Approval Study 
Protocol, References for tables provided in Package Insert 


9/22/11 


MELA 


Approvable Letter 


9/22/1 1 


FDA 


PMA Amendment P090012/A017 - In response to Approvable 
Letter: Revised Patient Labeling, Revised User Guide, Revised 
Package Insert, Answers to questions regarding Training 
Program and Revised Training Program elements, Responses to 
issues regarding the Post Approval Study and Revised Post 
Approval Study Protocol, Answers to questions regarding 
MelaFind software 


10/6/11 


MELA 


PMA Amendment P090012/A018 - Final Patient Labeling, User 
Guide, Package Insert, and Summary of Safety and Effectiveness 
Data 


11/1/11 


MELA 


PMA Approval 


11/1/11 


FDA 
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